Creative applications of computer vision
Neural style transfer is a two-source image-modification task: the target supplies content and the reference supplies style.
Two Images, One Generated Result
Neural style transfer is a two-source image-modification task. It starts with a target image and a reference image, then produces a generated image intended to combine information from both. The target supplies the subject and larger arrangement that should remain recognizable. The reference supplies visual qualities such as textures, colors, and patterns.
Neural style transfer is a two-source image-modification task in which the target image supplies content and the reference image supplies style.
Content and Style
The distinction between content and style is the central idea behind neural style transfer. Content describes higher-level image structure: the broad arrangement or recognizable subject. Style describes textures, colors, and visual patterns. Style can include patterns that appear at different spatial scales.
Separating the Two Contributions
Imagine that a target image supplies a recognizable subject and broad arrangement, while a reference image supplies distinctive textures, colors, and patterns. Identify what each image contributes to the generated result.
Inspect the target: Treat the target as the source of the subject and larger arrangement that should remain recognizable.
Inspect the reference: Treat the reference as the source of textures, colors, and visual patterns.
Describe the result: The intended generated image combines the target's content with the reference's style.
The subject and broad arrangement come from the target, while the visual appearance comes from the reference.
Two Losses, Two Objectives
A loss function makes the desired result measurable. Neural style transfer uses two related objectives. The generated image should have a content representation close to the target image's content representation, and it should have a style representation close to the reference image's style representation.
| Objective | Compared with | Desired relationship |
|---|---|---|
| Content loss | Target image's content representation | Generated content representation is close |
| Style loss | Reference image's style representation | Generated style representation is close |
The two losses express different requirements for the same generated image.
The difficult part is defining style and content mathematically. Deep convolutional neural networks provide representations that allow these two aspects to be defined and compared. Once the representations exist, the losses can compare the generated image with the appropriate source image for each objective.
Minimizing the Combined Loss
Neural style transfer follows a general deep-learning pattern: describe the desired result with a loss function, then minimize that loss. In this case, minimizing the combined loss is intended to produce an image that satisfies both requirements at once: recognizable target content and reference-like style.
Reading the Optimization Goal
A generated image is being evaluated using both a content objective and a style objective. Explain what minimizing the combined loss is intended to change.
Check content: The generated image should move toward a content representation close to the target image's content representation.
Check style: The generated image should move toward a style representation close to the reference image's style representation.
Combine the goals: The loss expresses both requirements, so minimizing it is intended to improve the generated image with respect to both content and style.
The intended result is a generated image that keeps the target's recognizable content while adopting the reference's visual style.
Common Misreadings
Treating the target image as the source of both content and style.
Neural style transfer is defined as a two-source task. The target supplies content, while the reference supplies style.
Fix:
Assign the subject and broad arrangement to the target, and assign textures, colors, and patterns to the reference.Treating style as only a label for an artistic category.
In this context, style has a specific meaning based on visual properties, including patterns at different spatial scales.
Fix:
Describe style through textures, colors, and visual patterns.Using only one loss objective.
The desired result has two parts: closeness to the target's content representation and closeness to the reference's style representation.
Fix:
Consider both content loss and style loss as parts of the combined goal.Assuming that minimizing loss means copying one source image completely.
The combined objective is intended to balance target content with reference style.
Fix:
Explain the result as a combination of the target's content and the reference's style.
Check Your Understanding
A generated image is recognizable as the subject and broad arrangement of the target, but its textures, colors, and visual patterns resemble the reference. Identify which image supplied the content, which supplied the style, and what two objectives the loss function represents.
Hints
- Content refers to the higher-level structure or recognizable subject.
- Style refers to textures, colors, and visual patterns.
- The two objectives compare the generated image with different source images.
What do you think happens?
Before reading the answer, predict what the generated image is intended to preserve from the target and reproduce from the reference.
Reveal answer
Answer: The target's content and the reference's style
The target supplies the subject and larger arrangement, while the reference supplies textures, colors, and visual patterns.
Key Takeaways
- Neural style transfer modifies an image using two sources: a target image and a reference image.
- The target supplies recognizable content, including the subject and broad arrangement.
- The reference supplies style, including textures, colors, and visual patterns.
- Content loss compares generated content with target content, while style loss compares generated style with reference style.
- Minimizing the combined loss is intended to produce recognizable target content with the reference image's style.