Evaluating Creative Outputs With Human Review
AI suits creative tasks largely because of its probabilistic nature, which produces a range of candidate outputs for human selection.
Trial and Error in Creative Work
Creative production has always involved trial and error. A designer might sketch several layouts before one feels right, and a marketing team might write several headlines before choosing one. This article looks at what happens when AI takes over part of that generative work, especially for pictures and video. The central idea is that AI produces a spread of candidate outputs, and a human decides which candidate is worth keeping.
What do you think happens?
If you give a creative AI system the same request twice, do you expect it to return the same image or clip both times?
Reveal answer
No, it selects from a range of possible outputs.
The source explains that AI works well for creative tasks because of its probabilistic nature. Rather than returning one fixed answer every time, a probabilistic system selects from a range of possible outputs. A single request can yield many candidate images, layouts, or clips, and a human can choose or refine among them.
Why Creative Tasks Suit Probabilistic Output
The source gives a specific reason that AI fits creative work: its probabilistic nature. A probabilistic system does not have to settle on a single result. It draws from a range of possible outputs, and that spread is exactly what a creative process needs. When a creative team wants options, a system that can produce several different candidates from one request is useful.
| Property | Fixed answer system | Probabilistic system |
|---|---|---|
| Output for one request | One result every time | Selected from a range of possible outputs |
| Value for creative work | Little variety to choose from | Many candidate images, layouts, or clips to compare |
| Role of the human | Accept or reject the single result | Choose or refine among the candidates |
Contrasting a fixed answer with the probabilistic output the source describes for creative tasks
The practical consequence is the important point for this article. Variety is cheap to produce, and selection is where human judgment enters. The source does not explain the internal mechanics of probabilistic behavior in this passage and points forward to a later chapter for a fuller treatment. For now, it is enough to understand that the machine supplies options and the person supplies the decision about which option to use.
Where AI Is Applied to Creative Work
The source describes four areas where AI is applied to creative work: image generation, video generation, profile pictures, and advertising. The relationships among them are easiest to see as a map, with each application linked to what the source says about it.
Image generation is the use of AI to create images. The source names Midjourney as a leading example. It also names Adobe Firefly, but associates it with photo editing rather than creating images from scratch. Keeping those two roles separate helps when reading about the field: one product is named for generation, and the other for editing existing photographs.
Video generation is the parallel application for moving pictures. The source lists Runway, Pika Labs, and Sora as AI startups in this area. Image and video generation together form a broader pattern: the same creative-application logic is being extended from still frames to sequences of frames.
Tracing One Base Advertisement
The source describes a concrete pattern in advertising. A single base ad produces several variants, and each variant is adapted to a season or a location. The variants are then tested, and the one that works best for the audience is kept. The diagram below follows that description. Notice that it has three stages: branching into variants, reaching an audience, and keeping the winner.
The source does not describe how the test is run. The evaluation step is therefore shown only at the level of the source's claim: the variants are tested against an audience, and the one that works best is kept. Any detail about test design, such as how long it runs or which measures are compared, is outside what this source provides.
Following a Candidate Through Review
The sequence of generation, review, and selection can be followed step by step. The flowchart below shows the order in which these actions happen. The AI produces a set of candidates from a request, a human reviews them, and the human either keeps one or refines among them before a final choice is made.
Choosing Among Candidate Banners
A marketing team asks an AI tool for promotional banners for one product. The tool returns five different candidates. How does the team move from these candidates to a finished choice?
- 1Receive the candidates
Because the system is probabilistic, the five banners differ from one another rather than being five copies of one fixed answer.
- 2Review each candidate
A person looks at the candidates and judges which ones fit the purpose of the advertisement. The source places this judgment at the selection step.
- 3Choose or refine
The team may choose a candidate that already works, or refine one of them. The source says a human can choose or refine among the candidates.
- 4Keep the result
The chosen banner becomes the output for use. This is the point where human judgment has entered the process.
The AI supplied variety, and the team supplied the decision. The finished banner came from human selection among the generated candidates.
Mistakes When Reading Creative AI Output
Common mistake Treating a probabilistic system as if it returns one fixed answer
What it looks likeAssuming that regenerating a banner will give the exact same banner each time.
Why it failsThe source explains that a probabilistic system selects from a range of possible outputs rather than returning one fixed answer every time.
Do this insteadExpect a spread of candidates and plan for a review step that compares them.
Common mistake Assuming the first candidate is the finished output
What it looks likePublishing the first image the tool returns without looking at the others.
Why it failsThe source places human judgment at the selection step, where a person chooses or refines among candidates.
Do this insteadReview the candidates and select or refine before using any of them.
Common mistake Confusing image generation with photo editing
What it looks likeDescribing Adobe Firefly as a tool for creating images from scratch.
Why it failsThe source names Midjourney for image generation and associates Adobe Firefly with photo editing rather than creating images from scratch.
Do this insteadKeep the two roles separate when naming products.
Common mistake Presenting a belief as a measured result
What it looks likeStating that AI headshots are proven to improve job prospects.
Why it failsThe source presents the benefit of AI headshots as a belief among candidates, not as a measured result.
Do this insteadDescribe it as a belief that candidates hold, and do not treat it as established evidence.
Common mistake Assuming the source explains how ad variants are tested, when it only states that they are tested against an audience
Practice: Planning Variants for a Base Ad
A company has one base advertisement for a warm drink. Using the pattern from this article, sketch the variants you would create for two seasons and two locations. Then describe the three stages: creating the variants, testing them against an audience, and keeping the one that works best. For each stage, say whether the source treats it as a machine task or a human decision, and explain your reasoning in one or two sentences.
Hints
- Start with the base advertisement and branch it into seasonal variants and location-based variants.
- The source says AI can produce the variants and that they are tested against an audience to find the best one.
- Look back at where the source places human judgment: at the selection step, where a person chooses or refines among candidates.
Key Takeaways
- AI suits creative tasks largely because its probabilistic nature produces a range of candidate outputs, so variety is cheap and selection is where human judgment enters.
- The source names Midjourney for image generation, Adobe Firefly for photo editing, and Runway, Pika Labs, and Sora for video generation, while profile pictures and advertising are described by their uses rather than a named product.
- AI-generated profile pictures moved from being banned by Facebook in 2019 to being offered as a built-in tool by many social media apps in 2023.
- In advertising, one base ad can produce seasonal and location-based variants, which are tested against an audience so the best-performing one is kept.