Evaluating Fine-Tuned Large Language Models

Before any evaluation code is written, three things need to be true: Ollama is installed, a model has been downloaded, and the two can talk to each other.

Programming

Why the Runtime Comes Before the Evaluation

Evaluating a fine-tuned language model in Python depends on a working local runtime. Before any evaluation code is written, three things need to be true: Ollama is installed, a model has been downloaded, and the two can talk to each other. Confirming this first matters because it separates setup problems from problems in your evaluation code. If something fails later, you can tell whether the fault lies in the environment or in the code you wrote.

What do you think happens?

You run ollama run llama3 for the very first time, and then run it again the next day. Does the second run download the model again?

Yes, the model is downloaded on every run
No, the model is downloaded only on first use

Reveal answer

Answer:

No, the model is downloaded only on first use

The source says the model is downloaded automatically the first time the command runs, and later runs do not need to download it again. The first run is therefore slower and needs about 4.7 GB of free disk space.

Installing Ollama and Starting It

The installation route depends on your operating system. The official instructions are on the Ollama website at ollama.com. Follow the path that matches your system. Installing Ollama does not by itself make it available to the command line, however. Something must be running in the background before a terminal command can reach it. Both routes covered below end in the same state: Ollama is running and the command line can send requests to it.

Which installation step do I follow on my operating system, and where does the choice branch? The choice is made once, and then the model command comes after it.

Two Ways to Start the Ollama Service

The source describes two ways to get Ollama running so that the command line can reach it. In the first, you run ollama serve in a separate terminal window and leave that window open. The service stays alive as long as that terminal stays open. In the second, available on macOS, you start the Ollama application and let it run in the background. In this case you do not need to run ollama serve yourself.

How do the terminal commands and the Ollama background service connect, and what starts the service in each case? Both routes lead to the same running service.

Downloading and Running Llama 3 8B

Once Ollama is running, through either route, the next command you run is the one that downloads and uses the model. The command for the 8-billion-parameter Llama 3 model is ollama run llama3. Enter it in a terminal, not inside a Python session. The command is not a Python statement, and running it inside Python will not start the model.

bash

ollama run llama3

ollama run llama3

Notes on the code

ollama is the command-line tool
run tells it to start a model and use it
llama3 names the 8-billion-parameter Llama 3 model

What happens in order when I run ollama run llama3? The model is downloaded only on the first use, and the output appears after that.

Worked example

Planning the First Run

You have a laptop with 5 GB of free disk space and have just installed Ollama. You want to run Llama 3 8B for the first time. Is the disk space enough, and what should you expect?

1
Identify the storage requirement
The source says the 8-billion-parameter Llama 3 model takes up 4.7 GB of storage.
2
Compare with the free space
5 GB is greater than 4.7 GB, so the download should fit, though it leaves little room to spare.
3
Predict the first-run behaviour
The model is downloaded automatically on the first run, so that run takes longer than later runs.

Result

The 5 GB of free space is enough for the 4.7 GB model. Expect the first run to take longer because of the download, and expect later runs to start without downloading again.

Checking the Setup Before Evaluation Code

Before you write any evaluation code, confirm that the installation, the service, and the model all work from the terminal. Running the checks in this order means that if one fails, you already know which earlier step succeeded.

1
Confirm that Ollama is installed using the route for your operating system.
2
Make sure the service is running, either through the Ollama application in the background or through ollama serve in a separate terminal left open.
3
Run ollama run llama3 in a terminal and wait for the first download to finish.
4
Confirm that the terminal shows output from the model, which shows that Ollama and Llama 3 work from the command line.
5
Only after these checks pass, begin writing evaluation code in Python.

Mistakes Beginners Make During Setup

Common mistake
Assuming installation alone makes Ollama available to the terminal
Why it fails
The source says installing Ollama does not by itself make it available to the command line. Something has to be running.
Do this instead
Start the service through the Ollama application on macOS, or run ollama serve in a separate terminal and leave it open.
Common mistake
Running ollama run llama3 inside a Python session
Why it fails
The source says the model must be run from the terminal rather than a Python session.
Do this instead
Open a terminal window and enter the command there.
Common mistake
Skipping the Ollama setup prompt on macOS or Windows
Why it fails
The terminal commands used later depend on command-line usage being set up, which is what the Yes selection enables.
Do this instead
When the application asks whether to install command-line usage, select Yes.
Common mistake
Starting evaluation code before confirming the model runs
Why it fails
When evaluation code fails, the cause could be the environment or the code. Without a working runtime, the two cannot be separated.
Do this instead
Confirm that the model produces output from the terminal first, then write evaluation code.

Trying the Setup Yourself

EASY

Work through the setup on your own computer. First, identify your operating system and state which installation route applies. Then state which of the two service-start methods you will use and explain why. Finally, run ollama run llama3 and report what the terminal shows after the first download finishes.

Hints

macOS and Windows use the application route; Linux uses the command from the Ollama website.
On macOS you can choose between the application in the background and ollama serve in a separate terminal.
A successful run ends with output from the model in the terminal, not with an error about the service.

Your attempt (not saved)

Key Takeaways

Installation depends on your operating system: on macOS and Windows, open the Ollama application and select Yes for command-line usage if prompted; on Linux, run the installation command from the Ollama website.
Ollama must be running before the command line can use it, either through ollama serve in a separate terminal left open or through the Ollama application running in the background on macOS.
The command ollama run llama3 downloads the 8-billion-parameter Llama 3 model on first use, which takes up 4.7 GB, and must be run in a terminal rather than a Python session.
Confirm that Ollama and Llama 3 work from the terminal before writing evaluation code, so that setup problems can be separated from problems in your evaluation code.

More concepts starting with E