Fine tuning tutorial¶

Prerequisites

Basic familiarity with Python
Some familiarity with linear algebra
If you have used a LLM-based chat application it’d be beneficial
Some basic understanding of how a machine learning/deep learning model is trained at a general level

The lesson

Optional material

Quantisation

Reference

Overview¶

Adapting large language models

Large language models (LLMs) are now ubiquitous in many applications and a variety of both open- and closed-weight are available for many disparate tasks. However, it can be the case that an off-the-shelf model does not perform well in a specific task, or that we would like a smaller model focused on some specific use cases and avoid expensive, large general-purpose models. In these cases, we need to find a way to “tweak” the behaviour of a model to fit the requirements. A LLM can be adapted to a task in several ways: we can change the instructions given to the model, provide external information at inference time, or continue training the model on examples of the desired behaviour.

These approaches form a progression:

Prompt engineering -> Retrieval-augmented generation -> Fine-tuning

Moving from left to right gives us additional ways to influence the system. It also introduces more infrastructure, more data preparation, more evaluation work, and more chances for failure.

The central principle of this lesson is:

Start with the simplest intervention

Use the least complex approach that can be shown, through evaluation, to meet the requirements of the application.

Prompt engineering is therefore not merely a preliminary step before fine-tuning. For many applications, it is the appropriate final solution. Retrieval-augmented generation is useful when the model needs access to external or changing information. Fine-tuning becomes useful when the remaining problem is a persistent pattern of behaviour that instructions and retrieved evidence do not solve adequately.

_images/adaptation-ladder.png — Prompt engineering changes the instructions. RAG adds external evidence. Fine-tuning changes model parameters. Original figure created by VSC here.¶

Who is the lesson for?¶

The lesson is intended for researchers, research software engineers, and technical staff who want to understand how large language models can be adapted to specialist tasks.

The main focus is not on any particular model or software library (our example will be, though). Instead, the lesson develops a conceptual framework that can be applied when planning an LLM-based application or a fine-tuning experiment.

Learning objectives¶

By the end of the lesson, learners should be able to:

explain how text becomes a sequence of contextual token representations;
describe the roles of tokenisation, embeddings, attention, and feed-forward layers;
distinguish a problem with instructions from a problem with evidence;
choose between prompting, RAG, and fine-tuning for a concrete use case;
explain why fine-tuning is not a reliable substitute for document retrieval;
describe the difference between full fine-tuning and parameter-efficient fine-tuning;
explain the central idea behind low-rank adaptation;
design a baseline and evaluation before introducing additional complexity;
describe the main ways of evaluating a trained LLM and the limitations of each;
interpret benchmark results with appropriate caution.

Lesson schedule¶

A possible schedule for a three-hour lesson is:

Introduction                              10 minutes
How an LLM processes text                 20 minutes
Break                                     10 minutes
Choosing an adaptation strategy           40 minutes
Fine-tuning large language models         40 minutes
Evaluating large language models          30 minutes
Discussion and conclusions                15 minutes

The episodes can also be taught separately. The adaptation-strategy episode does not require a detailed mathematical understanding of attention, although the anatomy episode provides useful context.

Software and infrastructure¶

The conceptual parts of the lesson do not require a GPU. Exercises can be completed through discussion or using an existing model interface.

A later hands-on fine-tuning lesson may use the Hugging Face ecosystem, including Transformers, Datasets, TRL, PEFT, and Accelerate. Those implementation details are deliberately kept separate from the conceptual material in this lesson.

Fine tuning tutorial¶

Overview¶

Who is the lesson for?¶

Learning objectives¶

Lesson schedule¶

Software and infrastructure¶

See also¶