Using Ollama for Model Evaluation

Installation depends on your operating system: on macOS and Windows you open the Ollama application and select Yes for command-line usage if prompted, while on Linux you run the command from the Ollama website.

Programming

Why Setup Comes Before Evaluation Code

Evaluating a fine-tuned language model in Python depends on a working local runtime. Before any evaluation code is written, three things need to be true: Ollama is installed, a model has been downloaded, and the two can talk to each other. This article walks through each of those steps in order, so that when something goes wrong later, you can tell whether the problem is in your setup or in your evaluation code.

What do you think happens?

Before Python can evaluate a model through Ollama, what must be true on your machine?

Only Python needs to be installed
Ollama must be installed, a model must be downloaded, and the two must be able to communicate
The model must be trained again from scratch

Reveal answer

Answer:

Ollama must be installed, a model must be downloaded, and the two must be able to communicate

The source says that evaluating a fine-tuned model in Python depends on a working local runtime, which means Ollama installed, a model downloaded, and proof that the two can talk to each other. Checking this first separates setup problems from evaluation code problems.

Choosing an Installation Route

The installation route depends on your operating system. The official instructions are on the Ollama website at ollama.com. The source gives two paths. On macOS and Windows, you open the Ollama application you downloaded. If the application asks whether to install command-line usage, select Yes, because the terminal commands used later in this article depend on that setup. On Linux, you instead run the installation command shown on the Ollama website, rather than opening a desktop application.

Given your operating system, which installation path do you follow, and where does the command-line choice come in?

Starting the Ollama Service

Installing Ollama does not, by itself, make it available to the command line. Something has to be running. The source describes two ways to get the service running. Both routes end in the same place: Ollama is running, and the command line can send requests to it. Once that is true, the next command you run in the terminal is the one that downloads and uses the model.

How the Command Line Reaches the Service

It helps to see the two parts as separate components. The terminal is where you type commands. The Ollama service is what answers them. Your command is sent from the terminal to the service, and the service responds. Read the diagram below from top to bottom to see the order in which messages pass between the two components when you type ollama run.

When you type ollama run, which component sends the request, which component answers it, and in what order do the messages pass?

Downloading and Running Llama 3

With the Ollama application or ollama serve running in another terminal, you run the model from the command line, not from inside a Python session. The model used in this article is the 8-billion-parameter Llama 3 model. The command is shown below.

bash

ollama run llama3

ollama run llama3

Notes on the code

Type this in a terminal, not in a Python session
llama3 refers to the 8-billion-parameter Llama 3 model

What happens step by step between typing ollama run llama3 and getting a usable model: when is the model downloaded, and when does it start responding?

Worked example

Running the model for the first time

You have installed Ollama on macOS, started the application so it runs in the background, and now want to run Llama 3 for the first time.

1
Confirm Ollama is running
The application is open and running in the background, so ollama serve is not needed.
2
Check free disk space
The model takes up 4.7 GB of storage, so the first run needs at least that much free space.
3
Run the command in a terminal
Type ollama run llama3 in a terminal window, not inside a Python session.
4
Wait for the first download
The model is downloaded automatically on first use, so this first run takes longer than later ones.
5
Run it again later
After the download, later runs do not need to download the model again.

Result

Llama 3 is downloaded once and is ready to run from the terminal, which means the setup is ready for the checks in the next section.

Mistakes Beginners Make During Setup

Common mistake
Installing Ollama and assuming the terminal can already use it
What it looks like
You finish the installer and immediately type ollama run llama3, but nothing responds.
Why it fails
The source says installing Ollama does not by itself make it available to the command line. Something has to be running.
Do this instead
Start the service first, either with ollama serve in a separate terminal or by running the Ollama application.
Common mistake
Closing the ollama serve terminal and expecting the service to keep running
What it looks like
You start ollama serve, then later close that window and try to run the model.
Why it fails
The source describes ollama serve as a command you run in a separate terminal and leave open.
Do this instead
Keep that terminal window open while you work, or use the macOS application route instead.
Common mistake
Running ollama run llama3 inside a Python session
Why it fails
The source says the model command must be run in a terminal rather than a Python session.
Do this instead
Open a terminal, run the command there, and then return to Python once the setup is confirmed.
Common mistake
Skipping the command-line prompt on macOS or Windows
Why it fails
The source says the terminal commands used later depend on selecting Yes for command-line usage.
Do this instead
When the application asks about command-line usage, select Yes.
Common mistake
Starting a download without enough free disk space
Why it fails
The source says the model takes up 4.7 GB of storage, and the first run needs that much free space.
Do this instead
Check available disk space before the first run.

Checking the Setup Before Evaluation

Before writing any evaluation code, confirm three things in order. First, confirm that the service is running, either because the Ollama application is open or because ollama serve is running in a terminal. Second, confirm that the model is available, which you do by running ollama run llama3 once and waiting for the first download to finish. Third, confirm that the command line can reach the service, which the model run itself demonstrates when the terminal shows output from the model. If any check fails, go back to the step it depends on: a failed model run usually points back to the service, and a service that is not running points back to the start-up route you chose.

MEDIUM

You have just installed Ollama on Linux and run the install command from the Ollama website. Describe the next two things you need to do before your evaluation code can reach the model, and explain why the second one cannot be done inside a Python session.

Hints

Think about what the source says is required before the command line can use Ollama.
Look at where the source says the model command must be run.

Your attempt (not saved)

Key Takeaways

Installation depends on your operating system: on macOS and Windows, open the Ollama application and select Yes for command-line usage if prompted; on Linux, run the installation command from the Ollama website.
Installing Ollama does not make it available to the command line by itself; start it either with ollama serve in a separate terminal left open, or by running the macOS application in the background.
Run ollama run llama3 in a terminal, not a Python session. The first run downloads the 8-billion-parameter model, which takes up 4.7 GB of storage.
Confirm the service, the model, and the command line all work before writing evaluation code, so setup problems are not confused with evaluation code problems.

More concepts starting with U