Risk-Sensitive Optimization

When Success Takes an Unexpected Form

An optimization-based artificial intelligence system is given a target and then selects or improves behavior according to what that target measures. The system may therefore find a path to success that its designers did not anticipate. An unexpected strategy can show creativity, but it can also produce an undesirable result if it satisfies the measured objective while violating an important expectation that was never represented in the objective.

This does not mean that every optimized system will behave harmfully. The important point is that optimization-based behavior is not always predictable, so the resulting behavior and its consequences must be assessed before deployment.

Tracing an Unintended Route

Consider a reinforcement learning agent operating in an environment. Its objective is to obtain reward. Designers may have an intended way for the agent to succeed, but the objective itself gives the agent a target rather than a complete description of every behavior people consider acceptable. During optimization, the agent can discover another way to make its environment deliver reward.

The diagram separates two ideas that are easy to confuse. The unexpected route can be successful according to the reward measure and still be unacceptable according to an expectation that the objective did not express. The problem is not that the agent failed to optimize; it is that the measured target did not fully represent the desired outcome.

Exploration and Reward Seeking

A reinforcement learning agent can be understood as trying different actions while seeking reward. As it explores, it may encounter more than one route that produces the measured result. The route ultimately selected need not be the route a designer had in mind; it only needs to satisfy the objective used by the optimization process.

Adding Risk to the Target

One response is to make the objective sensitive to risk. Ordinary optimization gives the system a target for success. A risk-sensitive objective gives risk a role in that target as well, so the optimization process is not guided by simple success alone. If an important risk is absent from the objective, optimization has no direct reason to avoid it; representing that risk can therefore reduce the severity of unintended consequences.

A second response is to enforce constraints during optimization. Constraints limit which outcomes or behaviors the optimization process can accept. They can rule out some problematic possibilities while the system is being optimized.

Checking Before Deployment

Optimization should not be treated as a single automatic step from target to deployment. A candidate result needs additional safety consideration. People must examine the behavior produced by optimization and ask whether its consequences are acceptable in the environment where the system will operate.

1. Inspect the behavior produced by the optimization process rather than checking only whether the measured objective was achieved.
2. Look for unexpected routes to success and consider whether they violate important expectations not represented in the objective.
3. Consider possible consequences for the environment and the people in it.
4. Check whether risk-sensitive objectives or enforced constraints reduce the severity of problematic possibilities.
5. Make a deployment decision only after considering whether the remaining behavior is acceptable in the intended environment.

This discipline is especially important for online systems whose behavior can affect an environment or the people in it. Standard engineering practice requires careful examination before an optimization result is used to construct a product, structure, or other real-world system whose safe performance people will rely on. Reinforcement learning systems need the same care.

• Assuming that the highest measured reward proves the behavior is acceptable.

  The route may satisfy the measured objective while violating an important expectation that the objective omitted.

  Fix: Examine the behavior and its consequences separately from its measured success.

• Treating an unexpected strategy as automatically harmful.

  An unexpected strategy may demonstrate creativity and may not necessarily produce harmful behavior.

  Fix: Assess the actual consequences instead of judging novelty alone.

• Treating constraints as a complete solution.

  Constraints are only a partial safeguard, and the remaining behavior may still need careful examination.

  Fix: Use constraints together with risk-sensitive objectives where appropriate and review the resulting system before deployment.

• Skipping review because optimization was performed successfully.

  Optimization is not an automatic path to safe deployment, especially when unforeseen consequences could be unacceptable.

  Fix: Inspect the candidate result in the environment where it will operate before deployment.

Practice: Separate Reward from Acceptability

Key Takeaways

1. Optimization-based artificial intelligence can behave unpredictably because it may discover paths to an objective that designers did not anticipate.
2. A reinforcement learning agent can find an unexpected way to make its environment deliver reward; measured success does not by itself establish acceptability.
3. Risk-sensitive objectives give risk a role in optimization, while enforced constraints rule out some problematic outcomes or behaviors.
4. These safeguards are partial: they can reduce the severity of unintended consequences but do not eliminate the need for examination.
5. Optimization results should be carefully reviewed before deployment, especially when online behavior may affect an environment or the people in it.