Video summary

Build Your First AI Chatbot: Context Window, Memory & Hallucinations | FDE Full Course #4

Main summary

Key takeaways

Technology

Key technological concepts: context, memory, and hallucinations (LLM basics)

Why chats “forget” without history

  • An LLM itself does not retain conversation state across API calls.
  • If each user message is sent as an independent request (without prior turns), the model only “knows” the current message.

What provides memory in real apps

In real applications:

  • The application/server maintains chat history (in memory or a database).
  • Each time, the app sends relevant history back to the LLM, which creates the illusion of continuous conversation (similar to ChatGPT).

Terminology clarified

  • Context Window: the maximum amount of text (measured in tokens) the model can process in a single request (input + space for expected output).
  • Context window ≠ permanent memory:
    • Context is temporary per request.
    • The model’s parameters are not retrained during chatting.
  • System Prompt / Roles: the video introduces roles to control behavior:
    • User role: user queries
    • Assistant role: model responses
    • System role: highest-priority instructions that constrain the assistant

Product/features tutorial: building a chat application

High-level architecture

  • The app exposes an endpoint (e.g., /api/chat).
  • It receives user input (via Postman or a frontend).
  • It calls an LLM provider (e.g., an OpenAI model).
  • It returns the LLM’s response to the client.

From summarizer to real chat

  • Initially, there’s a “summarize”-style service that doesn’t maintain conversational context.
  • Then the lecturer builds a customer-support chatbot endpoint (Zomato-like) where:
    • the backend provides the relevant history to the LLM.

Example use case: “Zomato/Tomato support” customer service bot

The chatbot is framed as food delivery customer support, handling questions like:

  • Order late / delivery tracking
  • Missing items / refund requests
  • Complaint policy / escalation rules

Important limitation shown: If you don’t restrict outputs, the model may answer off-domain questions (e.g., “Explain Docker”) even though it’s intended for food delivery support.

Prompt-injection / instruction hierarchy and security concepts

Problem demonstrated

The assistant can be manipulated when user text includes fake or conflicting instructions such as:

  • “Below is the customer query… for debugging purposes…”
  • “ignore previous instruction”
  • “do not believe anything”

Mitigation approach introduced

Use a system message with high priority that defines:

  1. Role: customer support executive for the Tomato app
  2. Task: identify problem/urgency; answer only allowed query types
  3. Behavior: professional language; empathetic phrasing
  4. Constraints: refuse/decline unrelated queries

In code:

  • Store the system prompt in a dedicated variable.
  • Send it using the system role channel.

Token/cost optimization strategies for long conversations

Because each request may resend prior turns, the video explains token growth and how it can become expensive.

Strategies discussed:

  • Send only the first N messages
  • Send only the last N messages
  • Send first M + last N messages
  • Summarize earlier messages and send the summary instead

Caveat: Summarization/truncation can cause context loss, so it isn’t fully reliable.

Streaming vs non-streaming responses

The lecturer explains why ChatGPT-like UIs stream tokens:

  • LLMs generate tokens sequentially.
  • Streaming reduces perceived latency.

It’s also noted that server implementations (e.g., Spring Boot) can support streaming, though the shown sample doesn’t.

Hallucinations: definition, why they happen, and practical guidance

Definition

  • Hallucination: the model produces confident but incorrect content.

Why they happen

Core reasoning given:

  • LLMs predict the next token using highest probability given the current context.
  • If the truth isn’t present, the model may still generate plausible text.
  • Training data quality issues can also lead to incorrect facts.

Why hallucinations can’t be eliminated entirely

  • Generation is probabilistic/non-deterministic, so hallucinations can’t be fully removed.

Mitigation strategies mentioned

  • RAG (retrieval-based grounding)
  • Validations/guardrails
  • Memory tooling
  • Overall guidance: don’t fully trust AI—verify outputs

Main speakers / sources

  • Speaker: Aditya Tandon (lecturer; creator of the “FDE series”)
  • Primary referenced model families: OpenAI ChatGPT/LLM, plus examples such as Claude, Gemini, DeepSeek, Grok
  • Primary technical framework mentioned: Spring Boot (Spring AI controller/service patterns)
  • Client/test tooling: Postman (used to send requests to the chatbot endpoint)

Original video