By Mahesh Palan
Published: July 2025

🚀 Introduction

Imagine a world where AI reads clinical notes and decides whether a patient needs follow-up — and even books it for them automatically. That’s the vision I had when building my healthcare automation agent. In this blog, I’ll walk you through how I used BERT, PyTorch, Temporal, and FastAPI to create a production-grade AI agent that makes smart decisions from unstructured clinical notes.

💎 Problem Statement

Doctors write tons of free-text notes like:

“Follow-up with orthopedics in 6 weeks.”
or
“No follow-up necessary at this time.”

But how can software understand this and act accordingly?

My goal:
✅ Automatically analyze clinical notes
✅ Predict if a follow-up is needed
✅ Book an appointment
✅ Notify the patient

🛠️ Step 1: Training the Clinical Note Classifier

I used bert-base-uncased via HuggingFace with a simple classifier head.

class FollowUpClassifier(nn.Module):
    def __init__(self, num_labels=3):
        super().__init__()
        self.bert = BertModel.from_pretrained("bert-base-uncased")
        self.classifier = nn.Linear(self.bert.config.hidden_size, num_labels)

    def forward(self, input_ids, attention_mask):
        output = self.bert(input_ids, attention_mask)
        pooled_output = output.pooler_output
        return self.classifier(pooled_output)

Classes: follow_up, conditional, no_follow_up

Saved with:

torch.save(model.state_dict(), "clinical_bert.pt")
tokenizer.save_pretrained("tokenizer/")

🧠 Step 2: Predicting Follow-Up

I then wrote a utility to run predictions:

def predict_follow_up(text: str) -> str:
    # Load model/tokenizer
    # Preprocess text
    # Run prediction
    # Map to class labels
    return label

🤖 Step 3: Temporal AI Agent

Temporal lets you define workflows and activities like a business process.

Activities:

@activity.defn
async def analyze_notes(notes: str) -> str:
    return predict_follow_up(notes)

@activity.defn
async def find_slot(doctor: str) -> str:
    return {"dr_smith": ["2025-07-26T09:00"]}.get(doctor, [None])[0] or "No available slot"

@activity.defn
async def book_slot2(patient_id: str, slot: str) -> str:
    return f"Booked {slot} for patient {patient_id}"

@activity.defn
async def notify_patient(patient_id: str, slot: str) -> str:
    return f"Notified patient {patient_id} about appointment at {slot}"

Workflow:

@workflow.defn
class ScheduleWorkflow:
    @workflow.run
    async def run(self, patient_id: str, visit_notes: str, doctor: str) -> str:
        result = await workflow.execute_activity(analyze_notes, visit_notes)
        if result == "No follow-up needed":
            return "No follow-up needed."

        slot = await workflow.execute_activity(find_slot, doctor)
        await workflow.execute_activity(book_slot2, patient_id, slot)
        await workflow.execute_activity(notify_patient, patient_id, slot)
        return f"Booked and notified for {slot}"

🌐 Step 4: FastAPI Integration

I exposed the workflow via HTTP:

@app.post("/agent-schedule", response_model=BookingResponse)
async def schedule_with_agent(event: VisitEvent):
    result = await client.execute_workflow(
        ScheduleWorkflow.run,
        args=[event.patient_id, event.visit_notes, event.doctor],
        id=f"schedule-{uuid.uuid4()}",
        task_queue="agent-task-queue"
    )
    return BookingResponse(
        patient_id=event.patient_id,
        booked="booked" in result.lower(),
        slot="2025-07-26T09:00",
        message=result
    )

Test it with:

curl -X POST http://localhost:8000/agent-schedule \
  -H "Content-Type: application/json" \
  -d '{
    "patient_id": "P12345",
    "visit_notes": "Patient to follow up with orthopedics in 6 weeks.",
    "doctor": "dr_smith"
  }'

✅ Output

Sample positive result:

{
  "patient_id": "P12345",
  "booked": true,
  "slot": "2025-07-26T09:00",
  "message": "Booked 2025-07-26T09:00 for patient P12345..."
}

Or if no follow-up needed:

{
  "patient_id": "P12345",
  "booked": false,
  "slot": null,
  "message": "No follow-up needed."
}

🔍 Learnings

• ✨ Using BERT for unstructured text works well for medical notes
• ⚙️ Temporal gave me robust control over asynchronous workflows
• 📁 FastAPI made this easily accessible

💡 Future Work

• 🚀 Connect to real EHR systems using HL7/FHIR
• 🗒️ Add human review stage for conditional predictions
• 📊 Improve slot logic using real calendar APIs

🤘 Final Thoughts

From model training to automated booking, this project showed how machine learning, workflows, and automation can transform healthcare. Let’s build the future of health — one follow-up at a time.