OpenAI Fine-tuning vs Distillation - Free Colab Notebook

Summary

The video demonstrates how OpenAI’s distillation workflow can use a stronger model—such as GPT-4o or o1—to help fine-tune a smaller, faster, or cheaper model, such as GPT-4o mini.

In the example, “distillation” means generating answers with a stronger model and using those answers as training data for the smaller model. The presenter carries out the process using a Google Colab notebook and OpenAI’s fine-tuning interface.

Why fine-tune or distill?

The presenter describes three situations where fine-tuning may help:

• Domain knowledge: A model lacks knowledge about a specialized subject, and retrieval methods such as RAG have not produced good enough results. Fine-tuning may improve performance alongside retrieval.
• Answer format or behavior: A model knows the relevant information but does not reliably follow a desired format or response pattern. Prompt engineering is worth trying first, but it can be brittle across a broad range of questions. Fine-tuning may make the behavior more consistent.
• Cost or latency: A powerful model works well but is too slow or expensive for an application. Distillation can transfer some of its performance to a smaller, faster, cheaper model.

Fine-tuning and distillation

The video distinguishes pre-training from supervised fine-tuning:

• Pre-training generally uses very large amounts of raw or minimally processed text.
• Fine-tuning generally uses curated conversational examples, often formatted as questions and answers.

It then describes two broad training approaches:

• Data-based training: Prepare a dataset and train on it.
• Distillation: Use a stronger “teacher” model to help train a smaller “student” model.

Two types of distillation are covered:

• Black-box distillation: Ask the teacher model to generate responses, then use those responses as fine-tuning examples. This is the approach demonstrated and does not require access to the teacher’s internal weights.
• White-box distillation: Compare the teacher’s and student’s token probability distributions, then train the student to match the teacher, typically using backpropagation. This requires access to model internals or probability outputs, which the presenter says OpenAI does not expose for this purpose.

The presenter emphasizes that the demonstrated OpenAI workflow is effectively supervised fine-tuning on teacher-generated data.

Demonstrated workflow

The example uses touch-rugby rules, a subject the presenter describes as relatively obscure, to test whether a smaller model can improve.

1. Install the OpenAI Python package and configure access. Install the package in Colab and load the API key from Colab’s secrets.

2. Choose teacher and student models. GPT-4o is the teacher, and GPT-4o mini is the student.

3. Prepare test questions. Write ten questions about touch-rugby rules. The presenter notes that some were drafted with help from ChatGPT.

4. Evaluate the models before training. Query each model with a sports-rules expert system message. Use a low temperature for more consistent answers and a top-p setting of 0.9. In one example, GPT-4o correctly says a touch-rugby try is worth one point, while GPT-4o mini gives the five-point answer associated with regular rugby.

5. Generate and store teacher responses. Use the teacher model to answer the questions, then store the completions in OpenAI with metadata so they can be found later. Confirm that the data appears in the same OpenAI project used for the API key. The presenter uses ten examples, described as the minimum needed for the fine-tuning run in the demo.

6. Start fine-tuning with the generated dataset. Select the student base model and the stored dataset in OpenAI’s fine-tuning interface, then add a suffix to identify the resulting model. The presenter recommends using validation data for a more rigorous run but omits it in this small demonstration. Defaults are used because the presenter is unsure how to interpret some of the hyperparameter settings.

7. Evaluate the fine-tuned model. Copy the fine-tuned model’s name into the notebook and test it. It answers the training question correctly. The presenter cautions that this is expected because the question was in the training set and the run appeared to overfit.

Improving the process and reported results

For more reliable results, the presenter recommends a more systematic synthetic-data pipeline:

• Use source documents to build a varied set of relevant questions.
• Add relevant background information to those questions.
• Generate answers with a strong model and a low sampling temperature.
• Filter out answers that are not correct.
• Use an evaluation pipeline to measure answer quality, including an LLM judge that compares generated answers with ground truth.
• Keep training and evaluation data separate, and vary evaluation question wording so the test does not simply check memorization.

In a larger touch-rugby experiment, the presenter describes approximately 193 training examples and 22 evaluation examples. Across repeated evaluations, GPT-4o mini without fine-tuning scored around 8 out of 10. Fine-tuning it brought performance to roughly 17–18 correct answers out of 22.

GPT-4o without fine-tuning performed similarly to GPT-4o mini on this task. Generating the training data with GPT-4o rather than GPT-4o mini did not clearly improve the result when the answers were grounded in source documents. Fine-tuned GPT-4o also performed well, perhaps somewhat better, though the presenter says the results were within the uncertainty of the comparisons.

Additional notes

• OpenAI’s evaluation feature accepts data in JSONL format and is intended to help assess fine-tuning performance. The presenter finds that it lacks a straightforward way to compare generated answers against supplied ground-truth answers and expects the evaluation tools to evolve.
• The presenter notes that sharing evaluation results with OpenAI may avoid evaluation charges, but could involve submitting data for model training. Companies should consider the value and sensitivity of their data.
• Google’s Gemini fine-tuning tools are described as early and restrictive in the presenter’s tests, with unclear or inconsistent limits on input and output lengths.
• The presenter says they do not believe Claude offers fine-tuning at the time of recording.
• Fine-tuning is presented as an additional tool alongside retrieval-augmented generation and prompt engineering, not as a replacement for them.

Speakers and sources featured

• Speaker: The video’s narrator and presenter from Trelis Research.
• Models discussed or used: OpenAI GPT-4o (teacher and evaluation model), GPT-4o mini (student and evaluation model), o1, ChatGPT, and an LLM judge used in the larger evaluation process.
• Other model providers discussed: Google Gemini and Anthropic Claude.
• Data and tools discussed: Touch-rugby questions and rules documents; OpenAI’s API, fine-tuning interface, stored completions, and evaluation feature; Google Colab.