Today we celebrate a HUGE milestone as we share our 400th Issue 🚀🚀🚀

We have come really far since the first newsletter edition in 2018, and today we can celebrate many more achievements since then:

Our mission continues to be the same; let’s make production machine learning something we can all do well and do responsibly.

Thank you for being part of it, and bring on 400 more! We look forward to continue this great momentum towards the new phase of the Institute!

This week in ML Engineering:

Production ML Across 2015-2035

My talk covering 20 years of Production ML is now up on YouTube! I look back across the last 10 years of MLOps, and make some predictions for the next 10 years - many of which are playing out scarily well!

This talk was very close to my heart, as during the first half I do a really in-depth analysis on the evolution of tooling, practices and limitations all the way back to the early days of MLOps.

This journey has really been a fun one, and it is clear to me we are still at the very beginning, with challenges like ML monitoring not yet nearly solved.

In the second half, I cover some of the opportunities looking forward, which talk about the evoltuion towards LLMOps, new stacks emerging, faster and more autonomous services and patterns.

The message I keep coming back to is that the lifecycle of a model only really begins once it is in production; everything before that is preparation. I have to say I am excited for the many more years ahead!

Zalando on Agentic Engineering at Scale

Another win for Europe! Germany’s tech powerhouse Zalando has published one of the most detailed pictures of an agentic engineering rollout that we have seen this year. I have to say that I have personally had so much fun as part of this agentic engineering transformation - the state of engineering feels reinvigorating!

This showcases some of the value obtained from rolling out LLM-based PR reviews, with 33% of Zalando’s PRs now being auto-approved as low risk. We have also seen lead times on PR time decrease by 20-40% across more than 250 engineering teams.

A lot of the infrastructure has also been built in-house with open source technology; we have a central zLLM service powered by LiteLLM, which serves OpenAI, Bedrock and Vertex models to around 2,000 monthly active users (+ growing).

There are also some clear challenges that this brings, we are seeingcyclomatic complexity spiking as agent adoption grows, commit messages ballooning to 5,000 characters, and challenges on tool migration / transformation.

The best thing here is the great collaborations and contributions that have come about. There are weekly guild sessions and monthly trainings running at 120 to 150 participants, as well as hackathons and agentic pilots.

For production ML practitioners the takeaway is that the measurement and governance work is what makes an adoption story legible at all, and it is great to see a European company leading on publishing it!

Stealing Reasoning from Encrypted Traces

Super interesting, and pretty scary! Researchers Max Planck Institute, ELLIS and Snyk et. al. have found that the you can basically reverse engineer the encrypted reasoning from frontier labs like Anthropic and OpenAI!

Initially it may seem that there’s nothing to worry about, however this encrypted reasoning blob has been assumed to be safe, and likely is scattered across telemetry, logs, session dumps, etc which we just assumed was not unsafe; so many teams may now have a big problem.

To provide some intuition, you use a frontier lab model, it “reasons” before it answers. The labs don’t show you that thinking, but when you get the answer, you also get a sealed blob representing the hidden reasoning.

The blob is encrypted so you can’t read it, but it gets sent back with your next message so the model can pick up where it left off.

Further to this, the hidden reasoning is not the same as the answer. The answer is what the model chose to tell you, after whatever filtering and summarising sits between the two.

This means the reasoning can contain intermediate conclusions the model discarded, inferences it drew about you that it didn’t state, or content it decided not to surface.

So the blob can hold strictly more about your situation than you ever saw on screen. If you’d forwarded that blob to someone, you may have handed over more than the conversation.

This is probably going to lead into a large rearchitecture from foundation models, and it is interesting to see that we’re only seeing the tip of the iceberg as these systems evolve.

Hugging Face on Open Models

Who is actually shipping the open models these days? Hugging Face has published their state of open models report: Chinese labs set the monthly size ceiling in every month, running from 754B up to 2.78 trillion parameters.

The labs from United States stayed under 130B in five of the past seven months this year, falling behind on the open model releases.

The new trend is more and more permissive licensing; out of 178 Chinese releases above 20B, 59% shipped under Apache 2.0 and 22% under MIT, with none carrying non commercial restrictions.

This is great! Although it carries a geopolitical weight that is becoming more and more clear.

The distribution numbers are the ones that surprised me most, as the models under 1B account for 83% of all time downloads, while everything above 100B accounts for only 1%.

HugginfFace is of course only “one perspective” and the results would be biased to what they can se in their platform, but overall the report seems quite measured, especially as they had even caveats like these.

Cloudflare on MCP Traffic Control

We need to think seriously about MCP traffic. Cloudflare has shipped protocol level detection for MCP-source traffic, classifying requests by the MCP-Protocol-Version header dynamically to be able to take action.

Namely this gives more control to website owners, as there is now an ability to action on an is_mcp rule for allow and block policies.

This also gives richer insights about which servers are actually being reached, and the option to force everything through their portals.

Claudflare is still upfront about the limitations of this, since this needs TLS inspection which does not come for free, and cannot see local stdio servers or anything marked “do not inspect”.

It is interesting to see MCP governance arriving as ordinary network policy rather than as something bespoke, but that makes sense! It is clear that this is where a lot of the agent security work is heading.

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here’s a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin