We commit to evaluate our AI systems against the domain-specific consequences of their behaviour, and to ensure robust monitoring capabilities are in place for as long as the system operates, enabling Evaluation by Observability.
A system can score well on public benchmarks and still be wrong in ways that matter in its domain. Evals should be built from the consequences of each kind of error, including capability evals for what the system must do, safety evals for what it must not, and cost-weighted measures that reflect the asymmetry between a false alarm and a missed harm.
The same questions continue once the system is in production and should be answered with robust observability. Behaviour must be tracked against real traffic rather than curated test sets, and where relevant advanced monitoring applied, such as drift and outlier detection, as well as agent/model-as-judge patterns throughout the data flow of the systems.
01 — WHERE IT FAILS
Where it fails
Evaluation fails silently: the score stays green while the system and its conditions drift.
- Benchmark contamination and overfitting inflating measured capability.
- No telemetry, tracing or logging in place to see behaviour after launch.
- Drift in data, usage or hosted model versions that goes undetected after deployment.
02 — PRACTICAL CONTROLS
Practical controls
Controls should make evaluation continuous, with production behaviour feeding the same suite that gated the release.
- Build domain-grounded capability and safety evals with cost-weighted metrics.
- Monitor production behaviour with drift and outlier detection, and agent/model-as-judge patterns where relevant.
- Feed incidents and near misses back into the evaluation suite, so each release is tested against the failures the previous one produced.