How to Choose Large Language Models: A Developer’s Guide to LLMs

Overview

This video is a developer-focused guide on choosing and testing Large Language Models (LLMs). It emphasizes that the “right” model depends most on the specific problem you’re solving—not just benchmark rankings.

Key decision factors for choosing an LLM

• Use case first: Benchmarks/leaderboards can help, but the primary driver should be the task you need to solve.

• Trade-offs matter:

  • Higher “intelligence” often increases cost
  • Smaller models can reduce latency (faster speed) and lower costs

• Selection impacts accuracy, cost, and performance

• Proprietary vs. open-source:

  • SaaS/proprietary models (e.g., GPT) are easy for rapid prototyping
  • Many organizations need control, customization, and flexibility, which open-source models like Llama and Mistral provide

How the speaker evaluates models (reviews/analysis tooling)

Intelligence vs. cost analysis (benchmarks-based)

• The video references intelligence scoring derived from benchmarks such as MMLU-Pro (and similar evaluations).
• It notes you may not need very large/advanced models for simpler tasks at scale (e.g., millions of queries).

Chatbot Arena Leaderboard (UC Berkeley / ALM Arena)

• Ranked using over a million blind user votes.
• Provides a “vibe score” style ranking that reflects perceived strengths in areas like:
  • reasoning
  • math
  • writing
• Suggested benefit: benchmarks can sometimes be reverse-engineered, so community voting offers a more robust signal.
• Includes an interface to compare models side-by-side using the same prompt (example given with Granite 8B vs Llama 8B, prompting for a JSON customer response).

Open LLM Leaderboard

• Focuses on open-source foundation and fine-tuned models.
• Provides metrics + filters to match deployment constraints, such as:
  • available GPUs
  • local execution
  • real-time inference on mobile/edge
• Links to Hugging Face so you can find models/datasets and see how to run them.

Local testing and demos (tutorial-style)

1) Run models locally with Ollama

• Uses Ollama (open-source developer tool) to run LLMs on your own system.
• Supports:
  • chat
  • vision
  • tool calling
  • RAG-embedding models
• Demo includes:
  • Running a Granite 3.1 model already quantized/compressed for the machine
  • A quick sanity-check prompt: “Talk like a pirate”

2) Build RAG (Retrieval-Augmented Generation) with Open WebUI

• Uses Open WebUI as an interface for interacting with the local model (Ollama-backed) and potentially remote OpenAI-compatible models.
• RAG demo flow:
  • Attach enterprise data the model wasn’t trained on (example: a document about Marty McFly)
  • Use an embedding model + vector database to retrieve relevant passages
  • Provide citations in the answer to establish a “source of truth”
• Core point: RAG helps the model answer questions grounded in your own documents.

3) IDE coding assistant using Continue (local model)

• Describes Continue as an open-source VS Code / IntelliJ extension for AI coding help.
• Configured to use the locally running Granite model via Ollama.
• Capabilities shown:
  • Chat with the codebase
  • Explain files
  • Make edits
• Example workflow: ask for documentation/comments inline, and the agent proposes changes for approval/denial.

Overall takeaway / guidance

• Model evaluation should be use-case-driven
• Use multiple signals:
  • online/community signals (leaderboards like Chatbot Arena)
  • benchmarks/metrics
  • local testing with your own data
• Mentions hybrid approaches, such as combining a powerful model with a smaller on-device model.

Main speakers / sources

• Speaker: “I’ll show you as a developer…” (single presenter; name not provided in subtitles)
• Sources mentioned:
  • Chatbot Arena Leaderboard (by UC Berkeley and ALM Arena)
  • Open LLM Leaderboard
  • Hugging Face
  • Artificial Analysis (used for the model landscape comparison)
  • Ollama
  • Open WebUI
  • Continue (IDE extension)
  • Example models: GPT, Llama, Mistral, Granite 8B / Granite 3.1