Choosing an adaptation strategy - aka when NOT to fine-tune¶

When an LLM application performs poorly, fine-tuning is rarely the first or only possible response. The model may need clearer instructions, better context, access to external information, or a more suitable evaluation procedure.

Prompt engineering, retrieval-augmented generation, and fine-tuning address different sources of failure:

Prompt engineering:
    change the instructions and examples

Retrieval-augmented generation:
    supply external evidence at inference time

Fine-tuning:
    change model parameters through training

These methods can be combined, but they should not be introduced indiscriminately. Every additional component has to be developed, evaluated, operated, and maintained.

Questions

What can prompt engineering change?
When does a model need retrieval rather than better instructions?
What kinds of behaviour justify fine-tuning?
Why is fine-tuning a poor substitute for a document store?
How can we evaluate whether additional complexity is worthwhile?
When should we decide not to fine-tune?

Objectives

By the end of this episode, learners should be able to:

distinguish a problem with instructions from a problem with evidence;
explain what prompt engineering, RAG, and fine-tuning change;
recognize applications for which fine-tuning is unnecessary or inappropriate;
choose an initial adaptation strategy for a concrete use case;
define a baseline against which a more complex system can be evaluated;
separate retrieval failures from generation failures;
describe when RAG and fine-tuning may usefully be combined.

How can we influence a model?¶

The progression from prompting to RAG to fine-tuning represents increasing system complexity.

../_images/adaptation-ladder.png — Prompt engineering changes the instructions. RAG adds external evidence. Fine-tuning changes model parameters.¶

The progression should not be interpreted as a sequence that every project must complete. A prompt-only system is not an unfinished RAG system, and a RAG system is not an unfinished fine-tuning project.

The objective is to find the least complex system that meets the application’s requirements.

The central diagnostic question

Is the model failing because it does not know what to do, because it lacks the necessary evidence, or because it cannot reliably perform the required behaviour?

Those three cases point toward prompting, retrieval, and fine-tuning respectively. Real systems may contain more than one kind of failure, but the distinction provides a useful starting point.

In-context learning¶

In-context learning (ICL) is an advanced AI capability introduced in the seminal research paper Language Models are Few-Shot Learners (2020) which unveiled GPT-3. Unlike supervised learning (or, in other words fine-tuning], where a model undergoes a training phase with backpropagation to alter its parameters, ICL relies entirely on pretrained language models and keeps their parameters unchanged. … At its core, in-context learning works by conditioning a large language model (LLM) on a prompt that includes a set of examples (input/output pairs or in-context examples) typically written in natural language as part of the input sequence. These examples, often drawn from a dataset, are not used to retrain the model but are fed directly into its context window. This window shows the amount of text an LLM can process at once, acting as its temporary memory for generating coherent responses and is the part of the model that processes sequential input.

—J. Varughese, IBM

In-context learning can be acheived by several methods.

Prompt engineering: prompt templates, few-shot prompting, modifying system prompt
Retrieval Augmented Generation (RAG)
Model Context Protocol and tool-calling, in general (not covered here).
Memory system (also not covered here).

../_images/prompt_context.webp — In 2026, a good majority of AI engineering is just context engineering.¶

Image credit: Anthropic.

Prompt engineering¶

Prompt engineering changes the model’s input. It does not update the model parameters.

A prompt can specify the task, the intended reader, the required structure, the available evidence, and what the model should do when it cannot answer. It can also contain examples of suitable inputs and outputs.

Consider this prompt:

Summarize this paper.

A more useful version might be:

Summarize the supplied paper for research software engineers who understand
machine learning but are not specialists in this application domain.

Use the following headings:

- Research question
- Method
- Main result
- Limitations
- Reproducibility

Use at most 300 words. Do not invent information that is absent from the
paper. If reproducibility information is not reported, write "not reported in
the supplied text".

The second prompt reduces ambiguity. It defines the audience, structure, length, and handling of missing information. Breaking a task into different steps or giving examples of good and bad behaviour in the prompt can also be very useful in many cases.

When prompting is enough¶

Prompt engineering is often sufficient when the base model already has the required capability and all necessary information can be included in the context.

Examples include rewriting text for a known audience, extracting fields from a short document, producing a specified format, explaining supplied material, or generating a first draft that will be reviewed by a person.

If a model produces good content under the wrong headings, the first response should be to specify the headings: training a model to solve a problem that can be expressed in two lines of instructions is difficult to justify.

A generative model may produce different wording across repeated calls. If the application requires a rigid structure, first try explicit output constraints, examples, schema validation, and deterministic or low-temperature decoding.

Fine-tuning may improve consistency, but it should not be the first mechanism used to impose a simple format.

Retrieval-augmented generation¶

Retrieval-augmented generation (RAG) adds an external information-retrieval stage.

A simplified RAG system looks like this:

user question
    -> retrieval query
    -> relevant passages
    -> prompt containing question and passages
    -> generated answer

The model parameters are not normally changed. The model receives additional evidence as part of its input.

What problem does RAG solve?¶

RAG is appropriate when the model cannot answer reliably from its existing parameters and the required information exists in an external collection.

Typical collections include internal documentation, scientific papers, technical manuals, organizational policies, support records, and databases whose contents change over time.

Retrieval is also useful when an answer must be accompanied by inspectable sources. A model may have encountered a fact during pretraining, but its parameters do not provide a dependable citation mechanism. A RAG system can retain the identifiers of the retrieved passages and ask the model to cite them.

A basic grounded-generation prompt could be:

Answer the question using only the supplied sources.

If the sources do not contain enough information, state that the available
evidence is insufficient.

Cite the source identifier after every factual claim.

Question:
{question}

Sources:
{retrieved_passages}

This prompt does not by itself create a reliable RAG system. The documents must be indexed, the query must retrieve useful passages, and the selected context must preserve the information needed to answer the question.

When RAG is worth introducing¶

RAG is a strong candidate when the application depends on information that is private, frequently updated, too large to include in every prompt, or expected to be cited.

It is less useful when the problem contains no substantial knowledge-retrieval component. If the task is to convert a known input into a fixed schema, adding a vector database may introduce infrastructure without addressing the actual failure.

RAG does not eliminate hallucinations (but can help greatly!)

Retrieval can ground the response in external evidence, but the model can still misread, ignore, or go beyond that evidence.

A grounded system needs explicit instructions, suitable retrieval, source tracking, and an evaluation that checks whether claims are supported.

Fine-tuning¶

Fine-tuning continues training a pretrained model on examples chosen for a particular objective. It changes some or all of the model parameters (or, sometimes, introduce a few new ones altogether).

This makes fine-tuning fundamentally different from prompting and retrieval: a prompt or retrieved document can be replaced immediately, while a parameter update is produced through a training process and is less directly inspectable.

Fine-tuning is most useful when the remaining problem is a repeated, measurable pattern of behaviour. Examples include using a specialized output representation, mapping recurring inputs to recurring outputs, or following domain/style conventions that are cumbersome to demonstrate in every prompt.

Fine-tuning may also teach a model to use specialist vocabulary more consistently.

When not to fine-tune¶

Fine-tuning is often proposed too early. Before training, it is worth checking whether the proposed use case matches any of the following situations:

The model needs access to policies, publications, software documentation or other information that changes in time (not too fast, hopefully). In these cases, RAG is usually a better place to start: injecting a lot of new information by updating model weights is not an effective update mechanism.
Traceability and explainability are required: once more, RAG is the better route.
The model is prompted in a vague way (e.g. “Summarise this”). Since the model knows nothing about what the user actually want, the first (and often only) step is to improve (a lot) the prompt.
The problem is with constraints, e.g. we want the output to be JSON only. In these cases, prompt engineering (possibly with examples) is paramount. If performance is still unsatisfactory, fine-tuning can improve format adherence, especially at scale and with smaller models.
There is no evaluation set: the only way to judge whether the fine-tuned model behaves better than baseline is to test on a hold-out evaluation set, or with user A/B testing in production. Without metrics for success, fine-tuning becomes a useless exercise (and may even lead to loss of performance in other general tasks).
There are not enough representative examples, since repetitive examples can easily lead to overfitting.

A practical decision process¶

The adaptation decision begins with a failure analysis rather than a choice of technology.

Establish the task and evaluation¶

Write down what a satisfactory output looks like. Assemble examples that cover ordinary cases, difficult cases, ambiguous cases, and cases where the system should decline to answer.

The evaluation need not initially be large. It does need to be representative enough to expose whether a change solves the intended problem.

Build a prompt baseline¶

Create the simplest prompt that specifies the task, context, constraints, output, and abstention behaviour. Include examples where they clarify the desired mapping.

Run the evaluation set and record the results. This baseline provides evidence against which retrieval or fine-tuning can be compared.

Add retrieval if evidence is missing¶

Inspect failed examples. If the model could answer correctly when given the right document passage, the problem points toward retrieval.

Evaluate whether the retrieval stage finds that passage. Do not evaluate only the final prose.

Consider fine-tuning for persistent behaviour¶

Fine-tuning becomes reasonable when the system has appropriate instructions and evidence but still fails in a repeated, measurable way.

At that point, the training examples can target a stable behaviour rather than compensating for an undefined prompt or missing source.

The resulting process is:

define success
    -> build a prompt baseline
    -> inspect failures
    -> add retrieval if evidence is missing
    -> inspect failures again
    -> fine-tune if behaviour remains inadequate

Combining RAG and fine-tuning¶

RAG and fine-tuning are not mutually exclusive.

A useful division of labour is that RAG supplies the evidence, whereas fine-tuning teaches the model how to use it.

For example, a technical assistant may need retrieval to access the current software documentation, but may also benefit from fine-tuning if it systematically fails to express commands according to an organization’s conventions.

It is worth noting that the combined system is more complex than either component alone. Thus, its evaluation must determine whether a failure came from retrieval, prompting, the fine-tuned behaviour, or the interaction between them.

Evaluating the adaptation ladder¶

Evaluation should precede an increase in complexity: otherwise, there is no reliable way to tell whether the additional component helped.

Evaluate more than fluency¶

A response can sound better without becoming more correct. Evaluation should therefore separate dimensions that matter to the application.

For a document question-answering system, useful dimensions might include factual support, citation correctness, completeness, appropriate abstention, latency, and cost.

For a structured extraction task, useful dimensions might include field accuracy, schema validity, handling of missing values, and robustness to unusual inputs.

Evaluate RAG in parts¶

A RAG system has at least three evaluation targets:

Retrieval:
Did the system retrieve the evidence required to answer?

Generation:
Did the model use the retrieved evidence correctly?

End-to-end:
Did the user receive a satisfactory answer?

A good final answer does not prove that retrieval is robust. The model may already know the answer or may have guessed correctly. Conversely, a poor final answer does not prove that retrieval failed. The correct source may have been present but used incorrectly.

Evaluate fine-tuning against meaningful baselines¶

A fine-tuned model should be compared with the unchanged base model using the best prompt developed during exploration.

Depending on the application, the comparison may also include:

base model with a simple prompt
base model with the strongest prompt
(if RAG is used) base model with prompt and retrieval
(if RAG is used) fine-tuned model with the same retrieval

This prevents a training run from receiving credit for improvements that came from a better prompt, a changed dataset, or a different retrieval setup.

Held-out evaluation is essential: examples used during training cannot show whether the model generalizes to new inputs.

Automated and human evaluation¶

Automated checks work well when correctness can be defined precisely. Schema validation, exact field comparison, classification scores, citation matching, and retrieval recall are useful examples.

Human evaluation is needed when quality depends on context, usefulness, or specialist judgment.

Another language model can assist with evaluation (LLM-as-a-judge), but its judgement must be also tailored to human preference. However, it can be useful in cases when the objective function is somewhat subjective, such as verbosity, complexity or general style. The section on evaluating large language models describes these approaches and their limitations in more detail.

Exercise: Choose the first intervention¶

Choose an adaptation strategy

For each scenario, decide what you would try first:

prompt engineering;
prompt engineering with RAG;
fine-tuning after establishing a prompt baseline;
RAG followed, if necessary, by fine-tuning.

Explain what kind of failure you expect the chosen approach to address.

A model must answer questions about policies that change every month and cite the relevant section.
A model writes useful summaries but uses the wrong headings.
A model must convert thousands of recurring incident reports into a strict internal representation.
A support assistant must use the current documentation and express commands according to an organization’s conventions.
A model must answer questions about a collection of papers that is updated every week.

exercise-adaptation-strategy

For the monthly policies, start with prompt engineering and RAG. The dominant problem is access to changing, citable evidence.

For the incorrect headings, start with prompt engineering. The desired change is simple to specify and should not initially require training.

For recurring incident reports, establish a prompt baseline first. Fine-tuning may become worthwhile if the mapping is stable, the volume is high, representative examples are available, and prompt-based output remains insufficiently reliable.

For the support assistant, use RAG to provide current documentation. If a strong prompt does not produce the required command style consistently, fine-tuning may then be justified for the behavioural part of the task.

For the changing paper collection, start with prompt engineering and RAG. The collection should be updateable independently of the model parameters.

Exercise: Decide whether to fine-tune¶

Fine-tuning readiness review

A team proposes fine-tuning a model on 200 internal question-answer pairs.

The stated goal is:

We want the model to know our documentation and stop hallucinating.

Before approving the training work, write at least five questions that the team should answer.

Identify which answers would suggest that fine-tuning is premature.

exercise-finetuning-readiness

Useful questions include:

How often does the documentation change?
Must answers cite a document or section?
Has the team evaluated a prompt-only baseline?
Has the team evaluated a retrieval-based baseline?
Are the 200 examples representative of real questions?
Is there a held-out evaluation set?
What does “stop hallucinating” mean in measurable terms?
Do failures occur because evidence is missing, or because the model uses available evidence incorrectly?
What should the system do when the documentation does not contain an answer?
Who will maintain and evaluate the model when documentation changes?

Fine-tuning is premature if the documentation changes regularly, citations are required, retrieval has not been tested, the same examples will be used for training and evaluation, or success has not been defined.

The phrase “know our documentation” points primarily toward retrieval. The team may later fine-tune how the model responds to retrieved material, but retrieval and abstention should be addressed first.

Summary¶

Prompt engineering, RAG, and fine-tuning change different parts of an LLM application.

Prompt engineering changes the instructions and examples. It is the correct starting point when the model already has the necessary capability and evidence.

RAG retrieves external information and includes it in the model’s context. It is the usual choice for private, changing, extensive, or citable knowledge.

Fine-tuning changes model parameters. It is best suited to stable and repeated behavioural patterns that remain inadequate after a strong prompt baseline and, where necessary, reliable retrieval.

The progression is therefore not:

start simple -> eventually fine-tune

It is:

start simple
    -> evaluate
    -> diagnose the remaining failure
    -> introduce only the component that addresses that failure

Fine-tuning is usually the wrong response when the model needs current documents, citations, clearer instructions, a simple output constraint, or a working retrieval system. It is also premature when there is no representative training data or held-out evaluation set.