Creating an Effective Project Showcase

The eval set is documented in /evals, with a README inside that directory.

Programming

Two Projects That Look Finished

Imagine two students who each present a project that uses a system to produce answers. Both projects run. Both produce impressive output. On the surface, both look finished. A showcase that stops at the surface cannot tell them apart, and that is the problem this article addresses. The difference that matters often lies in one section that many learners skip: the eval set.

What do you think happens?

Both projects run and both produce impressive output. What single thing decides whether a project is a demo or FDE work?

How impressive the output looks
Whether the project runs without errors
Whether the eval set is documented in /evals with a README
How many files the project folder contains

Reveal answer

Answer:

Whether the eval set is documented in /evals with a README

The source states that without documentation of the eval set, the project is a demo. With that documentation, the same project is FDE work. Running and looking impressive are shared by both projects, so they do not decide the question.

This article focuses on that one idea. It explains what the eval set documentation is, where it lives, what its README must cover, and how to check whether a project has it. The goal is to help you treat the eval set documentation as a required part of the showcase rather than an optional extra written at the end.

Following One Project as Documentation Is Added

The following scenario is an original illustration of the rule in the source. Follow the state of one project as each piece of documentation is added or left out. The project is a small system that produces answers. Its code works, and its outputs look good when run.

1
State 1: The project runs and the outputs look impressive. There is no /evals folder and no eval set documentation. The project is a demo. A reviewer who asks how the answers were judged has no place to look for the answer.
2
State 2: The student adds an /evals folder and a README inside it, but the README only says that the system was tested and that the results were good. It does not describe how the judging was done, where the test material came from, how labeling was performed, or what changed between versions. A reader still has to guess about most of the evaluation, so the project remains a demo in practice.
3
State 3: The student rewrites the /evals README so that it documents the rubric, the data source, the labeling protocol, and the version history. Now a reviewer can go to /evals, read how the evaluation was built and judged, and trust the reported results. The same project, with the same code and the same outputs, is now FDE work.

Notice that the code and the outputs did not change between the three states. Only the documentation changed. That is the central lesson: the eval set documentation is the difference, not the system's appearance.

Why the Documentation Separates FDE Work from a Demo

The source states the rule directly: without documentation of the eval set, a project is a demo, and with it, the project is FDE work. A demo shows that something can run and produce output. FDE work also shows how that output was checked. The README is where that checking is made visible to someone who did not build the project.

A README that lists all four items lets a reader trust the results. A README that covers only one or two leaves the reader guessing about the rest. The source does not give a template for each item, so the important part is that each of the four is clearly present and understandable to someone who did not build the project.

What does a demo project have that an FDE project lacks, and what does the documented eval set add to each side?

Where the Documentation Lives

The location is part of the requirement. The source places the eval set documentation in /evals, with a README inside that directory. A reader of your project should be able to go to /evals and find the README without searching. Think of that directory as the single place where a reviewer checks how your evaluation was built and judged.

Where does the eval set documentation sit relative to the rest of the project folder, and what does the directory structure look like?

The Four Items the README Documents

The source names four things that the README documents. Each one answers a different question a reviewer will ask about your evaluation. Together they cover how success was judged, where the test material came from, how it was labeled, and how the evaluation changed over time.

The rubric: the standard by which outputs are judged.
The data source: where the material in the eval set came from.
The labeling protocol: how the eval set items were labeled.
The version history: how the eval set has changed over time.

Each of the four must be clearly present and understandable to someone who did not build the project. Presence alone is not enough. A heading that says Rubric with nothing under it does not let a reader trust the results.

Auditing a Project Folder

A showcase can be checked against the source's rule. Because the source says checking a showcase means confirming all four items are present in the /evals README, you can audit a folder by following the checks below in order. The flowchart shows the sequence and the decision at each step.

What sequence of checks should I run on a project's /evals folder to decide whether its documentation is complete or incomplete?

MEDIUM

A student's /evals folder contains a README. Its sections are titled Rubric and Version History. The Rubric section describes how outputs are scored. The Version History section lists two versions. There is no description of where the test material came from and no description of how it was labeled. Using the flowchart, decide whether the documentation is complete, and explain which item or items are missing and why the project would still count as a demo.

Hints

Walk through the four checks in order and mark each item as present or absent.
The source says all four items must be present in the README, not just some of them.
Think about whether a reviewer who did not build the project could answer the missing questions from the README alone.

Your attempt (not saved)

Mistakes Learners Make with the Showcase

Common mistake
Treating the eval set documentation as an optional extra written at the end
What it looks like
A student builds the whole system, then writes a single line about testing in the README just before submitting.
Why it fails
The source states that without this documentation the project is a demo, so the documentation is what makes the project FDE work rather than decoration added afterward.
Do this instead
Plan the four README items as part of the showcase from the start, and check them as a required part of the project.
Common mistake
Placing the documentation somewhere other than /evals
What it looks like
The eval set notes are placed in the main README or in a file buried inside the source code folder.
Why it fails
The source places the documentation in /evals, and a reader should be able to find it there without searching.
Do this instead
Create an /evals folder and put the README inside it.
Common mistake
Documenting only some of the four items
What it looks like
The README describes the rubric and the data source but says nothing about labeling or version history.
Why it fails
A README that covers only one or two items leaves the reader guessing about the rest, so the results cannot be fully trusted.
Do this instead
Check all four items against the list: rubric, data source, labeling protocol, and version history.
Common mistake
Assuming that a project that runs and produces impressive output is already complete

Summary of the Showcase Requirement

At a glance

Two projects can run and look finished, yet only the one with documented eval set information is FDE work, according to the source.
The eval set documentation belongs in /evals, with a README inside that directory, so a reader can find it without searching.
The README documents four items: the rubric, the data source, the labeling protocol, and the version history.
Checking a showcase means confirming that all four items are present and understandable to someone who did not build the project.