Additional Project Showcase Elements

The eval set is documented in /evals, with a README inside that directory.

Programming

Two Projects That Look Finished

Imagine two students each present a project that uses a system to produce answers. Both projects run. Both produce impressive output. Both look finished on the surface. The difference that matters for a project showcase often lies in a single section that many learners skip: the eval set. The source for this topic states it directly. Without documentation of the eval set, the project is a demo. With that documentation, the same project is FDE work.

What do you think happens?

Two projects both run and both produce impressive output. One includes documentation of its eval set and one does not. Which one is FDE work?

The project with no eval set documentation
The project with eval set documentation in place

Reveal answer

Answer:

The project with eval set documentation in place is FDE work. The project without it is a demo.

The source states that without documentation of the eval set, a project is a demo, and with that documentation the same project is FDE work. Running well and looking finished are not enough on their own.

This article focuses on that one idea. It explains what the eval set documentation is, where it lives, what its README must cover, and how to check whether a project has it. The goal is to help you treat the eval set documentation as a required part of the showcase rather than an optional extra written at the end.

What Separates a Demo from FDE Work

A demo shows that something can work. FDE work, as the source frames it, includes the documentation that lets someone other than the builder understand how the evaluation was constructed and judged. The source is direct on this point: documentation of the eval set is what moves a project from one category to the other. Its presence or absence is the deciding factor, not how polished the outputs look.

Key point

Checking a showcase means confirming that all four items are present in the /evals README. A project that runs and produces good output still has to pass this check to count as FDE work.

Where the Eval Set Documentation Lives

The location is part of the requirement. The source places the documentation in /evals, and the README sits inside that directory. A reader of your project should be able to go to /evals and find the README without searching. Think of the directory as the single place where a reviewer checks how your evaluation was built and judged.

Where does the eval set documentation sit relative to the rest of the project, and which folder and file contain it?

In the tree above, the eval set documentation is a direct child of the /evals folder, which is a direct child of the project root. A reviewer does not need to search through the rest of the project to find it. The source does not name any other folders, so the only structural requirement you can rely on is this one: /evals, with a README inside it.

The Four Items the README Documents

The source names four things that the README documents. Each one answers a different question a reviewer will ask about your evaluation. Together they describe the eval set as a whole. The diagram below shows the README at the top and the four items beneath it.

What are the four components that sit inside the eval set README, and how do they relate to the eval set as a whole?

Rubric: the standard used to judge the system's outputs.
Data source: where the evaluation examples came from.
Labeling protocol: how the examples were labeled or judged, so the labels can be understood and repeated.
Version history: how the eval set has changed over time.

A README that lists all four items lets a reader trust the results. A README that covers only one or two leaves the reader guessing about the rest. Each missing item is a question the reviewer cannot answer from the documentation alone.

Tracing a Project's Documentation State

The following scenario is an original illustration of the rule in the source. Follow the state of the project as each piece of documentation is added or left out.

Worked example

A Project's Eval Set Documentation Over Three Stages

A student has built a project that uses a system to produce answers and has evaluated it. At each stage below, decide whether the project is a demo or FDE work according to the source.

1
Stage 1: No /evals README
The project runs and produces output, but nothing in /evals documents the eval set. Under the source's rule, without this documentation the project is a demo, regardless of how good the output looks.
2
Stage 2: README lists only the rubric
The student adds a README to /evals that explains the rubric. A reviewer now knows how outputs were judged, but still cannot tell where the evaluation examples came from, how they were labeled, or how the eval set has changed. Three of the four required items are missing, so the project still does not meet the rule.
3
Stage 3: README covers all four items
The student adds the data source, the labeling protocol, and the version history to the same README. Each item is clearly present and can be understood by someone who did not build the project. The source's condition is now met, and the same project counts as FDE work.

Result

The project changed from a demo to FDE work without any change to the system itself. Only the documentation in /evals changed.

Key point

The system did not change between stages. What changed was the state of the documentation, and that state alone determined which category the project fell into.

Auditing a Project Folder for Gaps

When you review a project showcase, you can work through the documentation in a fixed order. The flowchart below sets out the checks. Each decision either moves you to the next check or identifies a gap that must be fixed before the showcase counts as FDE work.

What sequence of checks determines whether a project's eval set documentation is complete or has gaps?

Read the flowchart from top to bottom. The first check is whether /evals exists at all. If it does, the next check is whether a README is inside it. Then confirm that the README covers the rubric, the data source, the labeling protocol, and the version history. The final check asks whether a reader who did not build the project can understand each item. Any no answer leads to a documentation gap.

Mistakes When Documenting the Eval Set

Common mistake
Writing the eval set documentation at the end of the project, as an optional extra
Why it fails
The source presents this documentation as a required part of the showcase. Treating it as optional is the habit that leaves a project as a demo.
Do this instead
Plan the four items as part of the showcase from the start, and check them before presenting.
Common mistake
Putting the README somewhere other than /evals
Why it fails
The source places the documentation in /evals. A reviewer who looks in the expected place will not find it without searching.
Do this instead
Create the README inside /evals so that a reader can go there directly.
Common mistake
Covering only the rubric, because it feels like the most important item
Why it fails
A README that covers only one or two items leaves the reader guessing about the rest.
Do this instead
Check the README against all four items: the rubric, the data source, the labeling protocol, and the version history.
Common mistake
Naming each item without explaining it
Why it fails
The source requires that each item be understandable to someone who did not build the project. A heading with no explanation does not meet that requirement.
Do this instead
For each item, write enough that an outside reader can see what it is and how it applies to the evaluation.

Practice: Audit a Project's README

MEDIUM

A classmate's project has an /evals folder containing a README. The README explains the rubric and the data source, and includes a line saying the labeling was done carefully. It does not mention the version history, and the labeling protocol is referred to but not described. Using the flowchart, identify each check the project passes and the check at which it fails. Then state whether the project counts as FDE work under the source's rule, and list what the classmate must add.

Hints

Start with the question of whether /evals exists and contains a README.
The source requires all four items to be present, not just the rubric and data source.
Ask whether a reader who did not build the project could understand the labeling protocol from the README as written.

Your attempt (not saved)

Key Takeaways for Your Showcase

1
The source states that without documentation of the eval set, a project is a demo; with it, the same project is FDE work.
2
The eval set documentation lives in /evals, with a README inside that directory, and a reader should be able to find it without searching.
3
The README documents four items: the rubric, the data source, the labeling protocol, and the version history.
4
Checking a showcase means confirming that all four items are present in the /evals README and understandable to someone who did not build the project.

At a glance

Documenting the eval set is what separates FDE work from a demo, regardless of how well the project runs.
The documentation belongs in /evals, in a README inside that directory.
The README must document the rubric, the data source, the labeling protocol, and the version history.
A project's documentation can be audited by checking the location, the presence of all four items, and whether an outside reader can understand them.