Build A Reasoning Model From Scratch 5: Inference Scaling 2 (Logprob Scoring, Self-Refinement)

Summary

The video explains two inference-time techniques for improving language-model answers without additional training: scoring multiple answers and self-refinement. It also introduces token-level log probabilities, which can be used to score answers and provide a foundation for later discussions of reinforcement learning.

1. Inference-Time Scaling Recap

The presenter places these methods within a broader set of inference-time techniques:

• Chain of thought: Prompt the model to explain its reasoning step by step. This can help basic models solve math problems.
• Self-consistency: Generate multiple answers through sampling, then choose the most common answer. Temperature and top-p sampling can provide answer diversity.
• Self-refinement: Have the model critique and revise an answer, optionally using a scoring function to decide whether to keep the revision.

The video focuses on self-refinement and ways to score candidate answers.

2. Rule-Based Answer Scoring

When multiple answers agree on a final result, a scoring function can help choose which full explanation to present. The example scorer rewards:

• Correct formatting: A final answer in a box receives a bonus.
• Including a number: An answer with a detectable numerical result receives a smaller bonus if it is not boxed.
• Brevity: A length-based bonus favors shorter responses, on the premise that generating fewer tokens costs less.

The presenter emphasizes that this is an empirical heuristic, not a reliable test of correctness. Its weights can be adjusted, and an overly strong preference for brevity could discourage useful explanations. Such scores are best treated as a way to break ties or rank candidates—not as proof that an answer is correct.

3. Scoring Answers with Token Probabilities

A model can score an existing response by calculating how likely it considers that response given the prompt. This is an evaluation step: the model is not generating the answer; it is measuring the probability of text that has already been produced.

The process is:

1. Tokenize the prompt and response into token IDs.
2. Run the combined text through the model to obtain logits at each position.
3. Convert the logits into probabilities with softmax.
4. At each position, select the probability assigned to the actual next token in the text. This requires aligning inputs and targets with a one-token shift.
5. Multiply the selected token probabilities to obtain the probability of the full sequence.

The sequence probability can be tiny, especially for long responses, because it is the product of many values below one.

4. Log Probabilities and Answer-Level Scores

To avoid numerical underflow when multiplying many small probabilities, the video works in log space:

• Take the logarithm of token probabilities, or use PyTorch’s combined log_softmax operation.
• Add the selected token log probabilities instead of multiplying raw probabilities. The log of a product is the sum of the logs, and the resulting values are easier to handle numerically.
• Exclude the prompt tokens when comparing candidate responses, since the prompt is shared.
• Optionally average the response-token log probabilities so that longer answers are not automatically penalized by accumulating more negative terms.

A higher, or less negative, score means the model considers the response more likely. However, likelihood is only a confidence proxy: a model can be confident in an incorrect answer, so log-probability scoring is not a correctness checker.

5. Self-Refinement Procedure

Self-refinement uses the model to critique and revise its own answer:

1. Give the model the original problem and generate an initial answer.
2. Ask it to act as a careful reviewer and produce a short critique plus a correction plan, preferably in bullet points.
3. Provide the original problem, the initial answer, and the critique to the model.
4. Ask it to revise the answer using the critique.
5. Optionally score the revised answer and compare it with the current answer.
6. Keep the revision only if its score improves; otherwise, retain the previous answer.
7. Repeat for a chosen number of rounds, or stop when further revisions fail to improve the score.

The demonstration shows an initially incorrect answer being revised to the correct result. The presenter notes that having the same model generate and critique its answer is imperfect. A stronger or separately trained verifier could provide better critiques, but would require additional resources.

6. Evaluation Results and Takeaways

On the MATH-500 evaluation, the video reports:

• The base model’s baseline accuracy is 15.2%.
• For the base model, one round of self-correction without a scoring evaluator gives the best result among the tested approaches. The presenter describes the gain as about 10% over baseline. Additional rounds do not help reliably.
• The heuristic and average-log-probability evaluators do not consistently improve on that result.
• For the reasoning model, the baseline is 48%. One round of self-correction reaches 56%, and the heuristic-scored version reaches 57%. Average log-probability scoring performs poorly in this test, at about 48%.

The main lesson is that self-refinement can improve results, but it is not automatic or guaranteed. More iterations are not necessarily better, and a model’s own likelihood score may fail to identify its mistakes. The log-probability material is included in part because it will be useful later in the series’ discussion of reinforcement learning.

Speakers and Sources Featured

• Speaker: The video’s presenter and narrator, identified in the video context as Sebastian Raschka. No other speaking participants are apparent in the subtitles.
• Sources and tools mentioned: PyTorch; the MATH-500 benchmark; the video’s earlier code and extract_final_candidate function, with related code referenced on GitHub; and the DeepSeek Math V2 work, cited as an example of using a verifier or critique model.