Loss Functions in Deep Learning

Two Images, One Result

Neural style transfer begins with two different roles. A content image supplies the subject and arrangement that should remain recognizable. A style image supplies the visual character that the generated image should adopt. The goal is a third image that combines both roles: it preserves the content of the original image while taking on the style of the reference image.

Following the Two Comparisons

The loss function gives the system a way to judge the generated image. One comparison checks generated content against the original content image. A second comparison checks generated style against the reference style image. These comparisons are then combined into one objective. The generated image is adjusted so that both distances are driven down together.

Representations from a CNN

The comparisons are possible because a deep convolutional neural network produces activations at its layers. Instead of judging the images only as raw visual inputs, style transfer uses these activations as representations. Different layers provide different kinds of information, allowing the content and style comparisons to focus on the properties they are intended to preserve or adopt.

Why Upper Layers Preserve Content

Upper convolutional layers are used as a content representation because their activations capture more global and abstract information. This makes them useful for preserving what the image is about: its subject and arrangement. Fine visual details may change as the generated image adopts the reference style, while the higher-level information needed to recognize the original content remains the target of the content comparison.

Representing Artistic Style

Style loss does not ask whether the generated image copies the style image as a complete scene. It uses relationships among CNN feature activations to represent the reference image's visual character. The corresponding relationships in the generated image are compared with those from the style image. A smaller difference means that the generated image is closer to the reference style according to this representation.

A Complete Style-Transfer Trace

The important trace is the chain of responsibilities. The content image is not compared with the style representation, and the style image is not used as the target for content preservation. Each image is assigned a role, each role receives a representation, and the generated image is compared with the appropriate reference before the losses are combined.

Optimization as Repeated Correction

The generated image is optimized through repeated updates. After an update, the two comparisons can be checked again: does the generated image better match the original content in the content representation, and does it better match the reference style in the style representation? The objective is successful only when both distances are driven down together rather than improving one while ignoring the other.

Debugging the Objective

A useful debugging strategy is to find the first place where the intended result diverges. Start with the desired roles, verify that each role has a corresponding representation, confirm that the generated image is compared with the correct input for each part of the loss, and then verify that the content and style distances are combined into one objective.

• Treating the content image and style image as if they supplied the same information.

  The two inputs have different assigned roles: one supplies content and the other supplies style.

  Fix: Keep content comparisons tied to the original content image and style comparisons tied to the reference style image.

• Using the wrong representation for content preservation.

  Upper-layer activations capture more global and abstract information, making them useful as a content representation.

  Fix: Use the upper-layer representation when the goal is to preserve higher-level image content.

• Checking only one part of the objective.

  The objective must drive both content and style distances down together.

  Fix: Inspect both comparisons and confirm that they are combined into one objective.

• Assuming that a correct network representation alone guarantees the intended result.

  The generated image must be compared with the correct input for each part of the loss.

  Fix: Trace each comparison from its generated representation back to the appropriate original or reference image.

Practice the Role Assignment

Key Takeaways

1. Neural style transfer assigns content to one input image and style to another, then seeks a generated image that combines both.
2. Content loss compares generated content with the original content image.
3. Style loss compares generated style with the reference style image using relationships among CNN feature activations.
4. A deep convolutional neural network provides the layer activations used to define content and style representations.
5. Upper-layer activations capture more global and abstract information, making them useful for preserving the original subject and arrangement.
6. The combined objective succeeds only when content and style distances are driven down together.