Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

The Coding-Agent Harness Leaderboard

Which agent harness should you run your open model in? Here's a leaderboard of harnesses across all models and reasoning effort across Claude Code, Codex, Pi, OpenCode, Hermes, and a few more:

Joel Niklaus from HuggingFace burned through the tokens so we don't have to, and ran each harness on Terminal-Bench 2.1 and DeepSWE three times per task with 95% confidence intervals with the data public. On behalf of the community - THANK YOU!

OpenCode and Kimi Code tie for first at around 52%, while OpenClaw comes last at 29.7%; that's a gap of about 23 points from the harness alone! Pi is the cheapest at about $0.76 per solved task, against $4.46 for OpenClaw.

The best harness also changes with the model, with OpenCode leading on GLM-5.3-Flash, Hermes on Flint and Codex on Ripple. Swapping the harness flips a task's outcome 28.1% of the time, against 24.1% for just rerunning it.

It is interesting to see how much of what we call model performance is actually the harness, so this is definitely worth checking before switching models.

Pinterest on Loop Engineering for Agents

Software factories at Pinterest! They shared how they automated the tuning of their internal Analytics Agent, with a process they call "loop engineering":

Basically, the challenge Pinterest has is that their agent kept skipping steps it should follow (e.g. inspecting table metadata before writing SQL), and manual prompt edits only got one of these rules to 64% compliance. So they built paired test suites for each rule, one where the step is required and one where it should be skipped, labeled by an LLM and reviewed by humans.

Each iteration starts a fresh session with a short structured handoff from the previous rounds, drafts up to 4 edits to prompts, skills or tools, and races them on a small sample. A winner only survives if it fixes more cases than it breaks, isn't hard-coded to the tests and doesn't hurt the regression suite; edits are also capped at a few dozen lines, so the loop can't just rewrite the app to pass its own tests!

The first large run reached 89.3% compliance in 30 iterations with 4 kept edits, and iterations 21 to 30 added nothing. The most interesting finding for me is the noise, as running the same code twice flipped the verdict on 10.3% of cases and the aggregate score completely hid it.

For production ML teams this is a great reminder that agent evals need repeated runs before we trust a single number.

Stanford AI100 on the Scaling Era

Stanford published their 2026 AI100 report which evaluates the last 5-years of AI:

It was quite intersting in particularly the retro, as the panel admits the 2021 report was wrong to say "the recent dominance of deep learning may be coming to an end". They also list what 2021 missed, which is inference-time reasoning and AI coding tools becoming a major channel of software output.

On scaling they argue that high-quality data is scarce and frontier training costs are becoming prohibitive, quoting Sutskever that pre-training is "essentially tapped out". The next phase in their view is smaller and more efficient models with more reasoning at inference time, which also flips the cost pattern as inference is no longer the cheap step!

On software engineering they call AI-assisted coding possibly the fastest shift in programming history; they also warn that vibe coding can hurt maintainability in production, and that junior programmers may not progress to senior roles.

For production ML practitioners it is a long read, but it is a great snapshot of where the field thinks it stands.

Amazon on Synthetic-Only Tabular FMs

Amazon launches a new tabular foundation model! Mitra-v2 is trained on synthetic data generated from a new hybrid causal-graph prior where each edge can be a small neural network, a decision tree, a Gaussian process or an autoregressive process:

The Transformer backbone is unchanged from v1; the gains come from the new prior and from pretraining tables with about 10x the rows and 3x the columns.

They claim it leads the single-model field on TabArena, beats v1 on 32 of the 33 datasets where both ran, and finishes all 51 datasets within the one-hour budget, where v1 failed on more than a third of them.

It isn't quite zero-shot though, as by default it fine-tunes 8 copies per table and averages their predictions; but that's just how it needs to work as it requires context to make a prediction like most other models of this type.

The weights and fine-tuning code are out under Apache-2.0, which is great! So it is definitely worth checking out if you still default to gradient boosting on structured data.

Cloudflare on AI Pentest Orchestrators

Cloudflare published what they are seeing from LLM-driven pentest orchestrators hitting their customers, and the numbers are pretty wild:

The creator of ARTEX, the framework that kicked this off, took the repo down on October 8. That fragmented the ecosystem instead of stopping it, as there are now over 12,000 repositories and several hundred forks, and Cloudflare says ARTEX and its forks make up over 60% of the orchestrators they track.

They claim daily requests for these tools have grown 30x since October 1, with 50+ framework families scanning their customers from thousands of IPs in just 3 days!

The incident they walk through is a good reminder of the basics, as the entry point was a static, non-expiring API token from a CI/CD pipeline with no IP restrictions and no rate limits. From there the agents spun up Worker scripts named artex-* and fired 400 database dump requests in an hour; Cloudflare does note it was "lightly guided by the human operator".

The second half of the post is mostly a pitch for their own products, but for production ML teams running agents with real credentials, rotating CI/CD tokens is definitely worth doing this week.

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin