Loading and Inspecting Instruction Datasets

Before a language model can be fine-tuned, its training data has to be read into memory and checked. A dataset can load without any error and still have the wrong shape, the wrong column names, or a lopsided set of labels. Checking first is the difference between a training run you can trust and one you have to repeat.

Instruction datasets store each example as a small record with an instruction, an optional input, and an output. A function called format_input turns one such record into a single prompt string. Many records are instructions that stand alone, with nothing extra to work on, so the first question to settle is whether the formatted prompt should contain an Input section at all.

The formatter makes one decision, and that decision is the only place where the output can change shape. It tests whether the input field is non-empty. If it is, the formatter adds an Input header and the input text. If it is not, that part of the output is skipped. Nothing else in the function changes between the two cases. This single test is what lets one function handle both kinds of entry without a separate template for each.

The formatter builds its output in two halves and then joins them. The first half always appears. It contains a fixed opening sentence, then an Instruction header, then the text taken from the instruction field. The second half is optional. It begins with a blank line, then an Input header, then the input text, and it exists only when the input field is non-empty.

Because both halves are plain strings, joining them is simply addition. The response is not built inside format_input at all. The caller attaches it after the prompt is returned. This design lets the same function produce prompts for inference, where no response exists yet, as well as complete training text, where the response is attached.

The example dataset is a collection of text messages. Each message has a label that says whether it is ham, meaning not spam, or spam. The dataset was saved earlier as a tab-separated text file named SMSSpamCollection.tsv inside a folder called sms_spam_collection. A tab-separated file uses tab characters, rather than commas, to divide one field from the next.

Each argument has a specific job. The sep argument tells pandas where one column ends and the next begins. Because header is set to None, pandas does not treat the first line of the file as column names, so the names argument provides them: Label for the first column and Text for the second. Writing df on its own line displays the table in a Jupyter notebook. Outside a notebook, print(df) gives a text version of the same table.

A loaded DataFrame is a table with rows and columns. Its shape is written as rows by columns. The source reports the spam dataset as 5572 rows by 2 columns. The 2 matches the two named columns, Label and Text, that the read_csv call created. The 5572 counts the examples: each row is one text message together with its label.

Inspecting the shape and a few rows confirms that the load worked, before any deeper analysis. Only after the table looks right does it make sense to ask how the labels are distributed.

A classifier learns from examples of every class, so it matters whether one class has far more examples than the other. The df["Label"] part selects the single column of labels. The value_counts() method then counts how many times each distinct value appears in that column and returns the counts sorted from largest to smallest.

Read the output against the earlier prediction about the dataset. The source finds that ham is the far more frequent label. A large gap between the two counts is what the source calls a class imbalance, and that gap is the reason the next step, undersampling, is needed.

If the training data holds far more ham than spam, the examples the model sees are dominated by one class. The source's reason for checking the counts is direct: a classifier learns from examples of every class, so a lopsided set of labels is a problem to fix before training begins, not after.

Undersampling reduces the number of examples in the larger class so that the classes end up with equal counts. The source undersamples the dataset to include 747 instances from each class. The reasons it gives are simplicity and a preference for a small dataset, since a smaller dataset makes fine-tuning the LLM faster.

The source says it uses a listing to perform the undersampling. That listing is not included in the material available for this article, so no code for it is reproduced here. What the source does establish is the goal of the step and the target count of 747 per class. Undersampling is one of several ways to handle class imbalance. The source acknowledges that other methods exist but treats them as beyond its scope, and it points readers to appendix B for them.

When a formatted string looks wrong, the bug is usually in one of a few predictable places. Work through format_input in the same order it runs, and stop at the first step whose output differs from what you expected.