Live Marathon- FDE Project Live Implementation

Summary of the Live Marathon (FDE Project Implementation)

Session goal

The speakers run a live, end-to-end implementation of an AI Forward Deployed (FDE) style healthcare application: “Secure EHR Insight and Clinical Validator” — an app that lets healthcare professionals query patient history using an LLM, while enforcing privacy (HIPAA/PHI protection) and safety guardrails.

---

Core technological concepts + product features

1) AI/Healthcare problem framing (HIPAA-first design)

• Hospital data is assumed to be stored in an EHR system backed primarily by:
  • a relational database (core table data)
  • other storage types (e.g., PDFs / blob-like data)
• Key requirement: do not send raw PHI/PII to the LLM.
• The project enforces a privacy-first pipeline:
  1. Retrieve relevant patient records safely
  2. Redact/mask PHI/PII
  3. Apply guardrails to block unsafe medical instructions (e.g., prescribing medications)

---

2) Data + infrastructure setup (Phase 0)

• Uses a realistic EHR dataset: MIMIC (tabular CSV) (not real patient data; synthetic/anonymized).
• Sets up an AWS environment manually:
  • EC2 instance (Ubuntu)
  • Security Group rules (SSH + PostgreSQL access)
  • Elastic IP for stable addressing
• Installs and configures PostgreSQL 14 on EC2.
• Creates a new database/user, then injects the CSV into:
  • A PostgreSQL schema/table (e.g., patient_encounters—described as a “patient counters” concept)

---

3) Vector search inside the existing PostgreSQL (PGVector)

Instead of building a separate vector DB (e.g., Pinecone), the design keeps the client’s DB ecosystem:

• Installs pgvector as a PostgreSQL extension.
• Adds a vector embedding column to the encounters table:
  • clinical_embeddings with vector dimension = 768
• Why 768:
  • It matches the chosen embedding model’s output dimension.

Semantic retrieval approach optimization

• Uses a patient dropdown / patient ID selection to avoid scanning all rows.
• Retrieval becomes scoped to a selected patient, reducing the search space from millions to ~hundreds of rows.
• Semantic matching is performed via cosine similarity/distance in SQL using pgvector operators.

---

4) Embedding generation (Phase 2 / embeddings pipeline)

• Creates embeddings from a combined context string, not only doctor comments:
  • Combines multiple fields such as admission type, test name, drug name, severity/description, and doctor comments.
• Uses domain-specific sentence transformer embeddings (medical/biomedical tuned).
• Avoids re-embedding:
  • Only processes rows where the embedding column is NULL.
• Batching considerations:
  • In the live demo, embedding a small subset is fast, but the full dataset could take hours.

---

5) PHI/PII Redaction (“Microsoft Presidio”)

Before anything reaches the LLM:

• Performs text redaction using Microsoft Presidio.
• Adds custom recognizers/patterns (e.g., region-specific handling for IDs like SSN patterns).
• Demonstrated behavior:
  • Removes/obscures names, emails, phone numbers, SSNs, and other identifying details.
• Output is “safe context” suitable for LLM consumption.

---

6) Safety guardrails (“NVIDIA NeMo Guardrails”)

After redaction, the system prevents unsafe medical actions:

• A guardrails configuration written in Colang defines allowed/disallowed behaviors.
• Example policy:
  • If the user asks for prescribing drugs / medical advice, the bot refuses.
  • If the user asks for patient history retrieval (e.g., last records, diagnoses, summaries), it proceeds.

Testing examples

• “Last recorded dosage of X” → allowed (history retrieval)
• “Recommend a higher dosage/prescribe” → blocked with a refusal message

---

7) Backend APIs (FastAPI) + LLM orchestration

A FastAPI service is built with endpoints used by the UI:

• Clinical query API

  • Input: patient_id + user prompt
  • Flow:
    1. Build retrieval query (embeddings + cosine search)
    2. Redact PHI from retrieved context
    3. Run guardrails decision (approve/refuse)
    4. If allowed, call the LLM with:
       • safe clinical context
       • the user question

• Patient list API

  • Returns patient IDs for the UI dropdown

Notes:

• LLM provider used for guardrails in the demo: DeepSeek API
• UI streaming uses OpenAI-compatible base URL variables

---

8) UI (Streamlit) with conversation memory

• Streamlit app:
  • Left sidebar: patient selection dropdown
  • Main chat: user asks questions; assistant responds
• Implements chat history via Streamlit session state
• Calls FastAPI endpoints for:
  • fetching patients
  • sending chat queries
• Adds explicit labeling/constraints for healthcare-style responses (e.g., “not for diagnostic purpose”).

---

Live deployment (Phase 3 / productionization)

Docker-based deployment on AWS

Production goal: run the app reliably from any environment.

• Uses EC2 + Docker with a container image built from the repo.
• Manual deployment steps:
  1. Install Docker on a new EC2 instance
  2. Clone the GitHub repo
  3. Create a private .env with credentials (not committed)
  4. Create .dockerignore to keep the image lightweight
  5. Build the Docker image (docker build)
  6. Run the container (docker run -d)
  7. Expose Streamlit on port 8501
• Validation includes:
  • Confirm FastAPI endpoints are alive
  • Confirm Streamlit UI is reachable publicly via public IP + port

---

Key analysis / takeaways emphasized by the speakers

• HIPAA compliance isn’t only “use HIPAA data.” It must be enforced through:
  • Redaction before LLM access
  • Guardrails to prevent disallowed medical actions
• Performance/scale:
  • semantic retrieval should be optimized (e.g., patient ID scoping)
  • embedding ingestion is time-consuming; the demo uses partial precomputed embeddings
• FDE philosophy:
  • emphasize architecture, data integration, deployment, and client constraints—not only model fine-tuning.

---

Main speakers / sources

• Monal — senior data scientist (AI/computer vision/generative AI/data); led problem/solution/system SDLC parts
• Bappy Ahmed Baki — mentor; supported implementation and backend/production sections
• Krishna Academy / Krishna — host/organizer; provided framing, agenda, and production/deployment coordination