Prompt Styles for Instruction Data
A dataset entry for instruction fine-tuning is a Python dictionary with 'instruction', 'input', and 'output' keys.
Predicting the Second Entry
Suppose a dataset has already been loaded from a JSON file into a Python variable called data. You print one of its entries, and then you print the entry at index 999, which is a second example from the same dataset. Before you look at the output of the second print, stop and predict it. Think about what kind of Python object an entry with several labeled parts is most likely to be, and what you would expect to see inside it.
What do you think happens?
What kind of Python object do you expect the second printed entry to be, and which keys would you expect inside it?
Reveal answer
A Python dictionary with the keys 'instruction', 'input', and 'output'.
Each entry has several labeled parts, and a dictionary is the Python object that stores labeled parts under keys. The second entry uses the same three keys as the first, but its input has no content.
The idea of this article is that an instruction fine-tuning dataset is built from entries, and each entry is a small structured record. Reading that structure carefully is the first step before anything else about training or prompt formatting makes sense.
Reading the Three Fields
A dataset entry for instruction fine-tuning is a Python dictionary with three keys: 'instruction', 'input', and 'output'. Each key names one field, and the value stored under that key is the text that belongs to that field. The three parts have distinct roles even though they all sit inside one object. The instruction tells the model what to do. The input holds the material the instruction works on. The output holds the answer the model should learn to produce.
The example below is a generated illustration in the same shape as the source's entries. It is a spelling task: the instruction asks for a correction, the input holds the misspelled word, and the output holds the corrected word.
{'instruction': 'Correct the spelling of the word.', 'input': 'recieve', 'output': 'receive'}
Correct the spelling of the word.
recieve
receive- The braces mark the start and end of the dictionary.
- Each quoted word before a colon is a key, and the quoted text after the colon is its value.
- Keys and values are separated by colons, and pairs are separated by commas.
- Indexing with a key name returns only that field's value.
Read the diagram as a lookup table. Entry['input'] returns the word recieve, and entry['output'] returns the word receive. The key name is what identifies the field, so you select a field by naming it rather than by counting its position.
Why Entries Are Dictionaries
Because each entry is a dictionary, you read its parts by key rather than by position. This choice has a practical consequence. If you want the instruction of an entry, you ask for entry['instruction'], and the key name is what identifies the field. Keys are case-sensitive strings in Python, so the names have to match the template exactly to retrieve the right value.
When the Input Is Empty
The input field may occasionally be empty, and code that reads entries should handle that case. In the second entry printed in the opening, the same three keys appear, but the input has no content at all, so the instruction stands on its own. The generated example below shows a filled input next to an empty one, so you can see what changes and what stays the same.
Name the capital city of France.- The key 'input' is still present even though its value is an empty string.
- An empty string is treated as false in the if test, so the code falls back to the instruction alone.
How Entries Feed Fine-Tuning
Instruction fine-tuning means training a model on a dataset in which the input-output pairs are explicitly provided. The entries you have examined are those pairs: each one supplies an instruction and an input, along with the output the model should produce for them. The entries were extracted from the JSON file, which is why they are the working units of the training data.
Note the direction of the flow. Entries are not the trained model and not the output of training. They are the material the training is built from, which is why their structure matters before any training begins.
Prompt Styles as a Separate Choice
There are various methods to format these entries for LLMs. The formatting describes how an entry's text is arranged for the model, and different formats are often called prompt styles. Two examples named in the course material are the Alpaca and Phi-3 styles, which were used in training notable LLMs.
Because the dictionary structure stays the same across prompt styles, you can read and check an entry first, then decide later how it should be formatted. This article does not reproduce the exact template of each named style, so treat the names as labels for formats you will meet rather than as templates to memorize here.
Mistakes When Reading Entries
Common mistake Treating the entry as a list and reading fields by position
What it looks likeentry[0] applied to a dictionary
Why it failsEntries are dictionaries, so fields are read by key name, not by numeric position.
Do this insteadUse entry['instruction'], entry['input'], or entry['output'] to read each field.
Common mistake Assuming an empty input means the key is missing
What it looks likeChecking whether 'input' in entry returns True, but then assuming the input has text
Why it failsThe input key is still present even when its value is an empty string, so a presence check does not tell you whether the field has content.
Do this insteadCheck the value itself, for example with if entry['input']:, before using it.
Common mistake Mismatching key capitalization
What it looks likeentry['Output'] when the key is 'output'Why it failsKeys are case-sensitive strings, so a different capitalization names a different key.
Do this insteadCopy key names exactly as they appear in the entry.
Common mistake Confusing the dictionary structure with a prompt style
What it looks likeSaying the entry itself is the Alpaca format
Why it failsThe dictionary structure is how an entry is stored. A prompt style is a separate way of formatting that entry for LLM training.
Do this insteadDescribe the entry's keys first, then name the prompt style used to format them.
Practicing With a New Entry
Work through this new entry before using it: {'instruction': 'Translate the word to French.', 'input': '', 'output': 'chat'}. What does entry['input'] return, and what should code do before using that field?
Hints
- Look at the value stored under the 'input' key, not at whether the key exists.
- An empty string has no content, so the instruction is the only text to rely on.
- A check such as if entry['input']: separates entries that have input text from those that do not.
Key Takeaways
- A dataset entry for instruction fine-tuning is a Python dictionary with the keys 'instruction', 'input', and 'output', and each key names one field.
- Entries are read by key rather than by position, and key names are case-sensitive, so they must match exactly.
- The input field may be empty while its key remains present, so code that reads entries should check the value before using it.
- Entries are the input-output pairs that instruction fine-tuning trains on, and prompt styles such as Alpaca and Phi-3 are separate formats for arranging those entries.