We commit to assess the consequences of incorrect outputs and automated actions and to design systems with human oversight to ensure aligned and safe outcomes, enabling Human Augmentation by Oversight.
Irrespective of how many levels of abstractions are introduced through AI systems, the impact is and will always continue to be human. AI systems should be developed to augment human cognition and capability as a whole. Human oversight must be enabled, and where reasonable, with human-in-the-loop established to drive remediation to predictions in contexts of high or unacceptable risk outcomes. This is a requirement now codified in Article 14 of the EU AI Act, which the institute contributed to through its policy work.
The AI Act also draws the automation line in law: Article 5 prohibits some practices outright, such as social scoring and manipulative systems, while the high-risk uses of Annex III, including justice, healthcare and critical infrastructure, may only operate under the human oversight of Article 14. In these domains a single wrong prediction can carry generational impact, so the level of automation must follow the consequence, not the capability.
01 — WHERE IT FAILS
Where it fails
Oversight designed for single ML models does not transfer to modern AI & Agentic systems. An agent can take hundreds of actions in one task, so reviewing each one is impossible; however, in some contexts reviewing none may be negligent.
- Automation that displaces meaningful human judgement.
- Agents that operate outside the mandate their operators intended.
- Interfaces which conceal uncertainty, or review exists only on paper.
02 — PRACTICAL CONTROLS
Practical controls
Controls should keep accountable people close to consequential decisions and scale oversight to the system’s autonomy.
- Assess the consequences of errors before automating a decision.
- Bound what a system may do alone and gate consequential actions on approval.
- Measure outcomes for the people affected, and keep escalation paths that work.