Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐
If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!
This week in ML Engineering:
- The Coding-Agent Harness Leaderboard
- Pinterest on Loop Engineering for Agents
- Stanford AI100 on the Scaling Era
- Amazon on Synthetic-Only Tabular FMs
- Cloudflare on AI Pentest Orchestrators
- Open Source ML Frameworks
- Awesome AI Guidelines to check out this week
- + more 🚀
The Coding-Agent Harness Leaderboard
Which agent harness should you run your open model in? Here's a leaderboard of harnesses across all models and reasoning effort across Claude Code, Codex, Pi, OpenCode, Hermes, and a few more:
Joel Niklaus from HuggingFace burned through the tokens so we don't have to, and ran each harness on Terminal-Bench 2.1 and DeepSWE three times per task with 95% confidence intervals with the data public. On behalf of the community - THANK YOU!
OpenCode and Kimi Code tie for first at around 52%, while OpenClaw comes last at 29.7%; that's a gap of about 23 points from the harness alone! Pi is the cheapest at about $0.76 per solved task, against $4.46 for OpenClaw.
The best harness also changes with the model, with OpenCode leading on GLM-5.3-Flash, Hermes on Flint and Codex on Ripple. Swapping the harness flips a task's outcome 28.1% of the time, against 24.1% for just rerunning it.
It is interesting to see how much of what we call model performance is actually the harness, so this is definitely worth checking before switching models.
Pinterest on Loop Engineering for Agents
Software factories at Pinterest! They shared how they automated the tuning of their internal Analytics Agent, with a process they call "loop engineering":
Basically, the challenge Pinterest has is that their agent kept skipping steps it should follow (e.g. inspecting table metadata before writing SQL), and manual prompt edits only got one of these rules to 64% compliance. So they built paired test suites for each rule, one where the step is required and one where it should be skipped, labeled by an LLM and reviewed by humans.
Each iteration starts a fresh session with a short structured handoff from the previous rounds, drafts up to 4 edits to prompts, skills or tools, and races them on a small sample. A winner only survives if it fixes more cases than it breaks, isn't hard-coded to the tests and doesn't hurt the regression suite; edits are also capped at a few dozen lines, so the loop can't just rewrite the app to pass its own tests!
The first large run reached 89.3% compliance in 30 iterations with 4 kept edits, and iterations 21 to 30 added nothing. The most interesting finding for me is the noise, as running the same code twice flipped the verdict on 10.3% of cases and the aggregate score completely hid it.
For production ML teams this is a great reminder that agent evals need repeated runs before we trust a single number.
Stanford AI100 on the Scaling Era
Stanford published their 2026 AI100 report which evaluates the last 5-years of AI:
It was quite intersting in particularly the retro, as the panel admits the 2021 report was wrong to say "the recent dominance of deep learning may be coming to an end". They also list what 2021 missed, which is inference-time reasoning and AI coding tools becoming a major channel of software output.
On scaling they argue that high-quality data is scarce and frontier training costs are becoming prohibitive, quoting Sutskever that pre-training is "essentially tapped out". The next phase in their view is smaller and more efficient models with more reasoning at inference time, which also flips the cost pattern as inference is no longer the cheap step!
On software engineering they call AI-assisted coding possibly the fastest shift in programming history; they also warn that vibe coding can hurt maintainability in production, and that junior programmers may not progress to senior roles.
For production ML practitioners it is a long read, but it is a great snapshot of where the field thinks it stands.
Amazon on Synthetic-Only Tabular FMs
Amazon launches a new tabular foundation model! Mitra-v2 is trained on synthetic data generated from a new hybrid causal-graph prior where each edge can be a small neural network, a decision tree, a Gaussian process or an autoregressive process:
The Transformer backbone is unchanged from v1; the gains come from the new prior and from pretraining tables with about 10x the rows and 3x the columns.
They claim it leads the single-model field on TabArena, beats v1 on 32 of the 33 datasets where both ran, and finishes all 51 datasets within the one-hour budget, where v1 failed on more than a third of them.
It isn't quite zero-shot though, as by default it fine-tunes 8 copies per table and averages their predictions; but that's just how it needs to work as it requires context to make a prediction like most other models of this type.
The weights and fine-tuning code are out under Apache-2.0, which is great! So it is definitely worth checking out if you still default to gradient boosting on structured data.
Cloudflare on AI Pentest Orchestrators
Cloudflare published what they are seeing from LLM-driven pentest orchestrators hitting their customers, and the numbers are pretty wild:
The creator of ARTEX, the framework that kicked this off, took the repo down on October 8. That fragmented the ecosystem instead of stopping it, as there are now over 12,000 repositories and several hundred forks, and Cloudflare says ARTEX and its forks make up over 60% of the orchestrators they track.
They claim daily requests for these tools have grown 30x since October 1, with 50+ framework families scanning their customers from thousands of IPs in just 3 days!
The incident they walk through is a good reminder of the basics, as the entry point was a static, non-expiring API token from a CI/CD pipeline with no IP restrictions and no rate limits. From there the agents spun up Worker scripts named artex-* and fired 400 database dump requests in an hour; Cloudflare does note it was "lightly guided by the human operator".
The second half of the post is mostly a pitch for their own products, but for production ML teams running agents with real credentials, rotating CI/CD tokens is definitely worth doing this week.
Upcoming MLOps Events
The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.
You can also find our upcoming events and past talk recordings on the talks and events page.
Other relevant events:
- MLOps World 2026 - Nov @ Austin
- WAICF 2027 - Feb @ Cannes
- AI Engineer Europe 2027 - Feb @ London
- KubeCon + CloudNativeCon Europe 2027 - March @ Barcelona
- QCon London 2027 - April @ London
- MLSys 2027 - May @ Santa Clara
- The AI Summit London 2027 - June @ London
- AI Engineer World's Fair 2027 - June @ San Francisco
- RAISE Summit 2027 - July @ Paris
- WeAreDevelopers World Congress 2027 - July @ Berlin
- AGNTCon + MCPCon Europe 2027 - Sept @ London
- HumanX EMEA 2027 - Sept @ Amsterdam
- World Summit AI Europe 2027 - Oct @ Amsterdam
In case you missed our talks, check our recordings below:
- Observability in the Age of AI Agents - Signals Berlin 2026
- The State of AI in 2025 - WeAreDevelopers 2025
- Prod Generative AI in 2024 - KubeCon AI Day 2025
- The State of AI in 2024 - WeAreDevelopers 2024
- Responsible AI Workshop Keynote - NeurIPS 2021
- Practical Guide to ML Explainability - PyCon London
- ML Monitoring: Outliers, Drift, XAI - PyCon Keynote
- Metadata for E2E MLOps - Kubecon NA 2022
- ML Performance Evaluation at Scale - KubeCon Eur 2021
- Industry Strength LLMs - PyData Global 2022
- ML Security Workshop Keynote - NeurIPS 2022
Open Source MLOps Tools
Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:
- SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
- KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
- Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
- Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
- AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
- Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain
Please do support some of our open source projects by sharing, contributing or adding a star ⭐
About us
The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.