Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

Netflix's Knowledge Graph for Agents

Knowledge bases are now one of the most valuable things we can build for our AI agents, but how do we start? Netflix just shared a great example of how they built an LLM knowledgebase for production systems.

Netflix basically built an ontology-driven knowledge graph that sits underneath their agentic root cause analysis system, which is then able to automatically and efficiently extract signal from their large scale production systems.

Their starting point is a home screen regression that took 4 hours to fix, and most of that time went into reconstructing how systems relate to each other; an application in one tool is a service in another and a cluster in a third, and only a few people carry that mapping in their heads.

Their created an is an ontology on the W3C stack (RDF, OWL, SHACL and SPARQL) stored in their own RDF graph database, with dozens of "enrichers" which turn telemetry+alerts+deployments+Slack channels into facts where every application gets validated against their software catalog of around 49,000 entries.

I found quite interesting how they make LLM output trustworthy, as they have LLMs summarising the incident discussions and using fuzzy matching to resolve the names, however nothing enters the graph unless it fits the type system, so the same query returns the same answer every time.

They are upfront that it is early, as the ranked hypotheses are sometimes wrong and there are no before/after numbers yet. For production ML practitioners the takeaway is that a lot of the agent quality comes from the context layer we give them!

Europe Tops Time Series Twice

Two European time series foundation models claimed the top spot in the same week, and I couldn't pick one so I'm sharing both 🇪🇺🚀🇫🇷 The Forecasting Company released t0-gamma, and two days later Mistral released Timestral, their first family of time series models.

t0-gamma follows the t0-beta release we covered a few weeks ago, and at release it ranked first among time series foundation models on GIFT-Eval, fev-bench and TIME. They use the same checkpoint everywhere with no task-specific retraining, and the weights are on Hugging Face.

Timestral comes in 4 sizes from 5M to 209M parameters under a modified MIT licence, and Mistral claims the 209M model ranks first on the same three benchmarks. Their charts put it narrowly ahead of t0-gamma on all 5 metrics (e.g. 50.4 vs 49.4 on fev-bench). It also runs on a CPU, at around 140ms for a day-ahead forecast with the largest model, and most of its training data was synthetic.

The margins are tiny and both results are self-reported, with Mistral's technical report still to come. Still, it is great to see two Paris teams at the top of a benchmark where TimesFM and Chronos were the names to beat!

Anthropic on Agent-Scale Code Review

The Claude Code team at Anthropic shared how they do code review now that Claude writes around 90% of their code:

They report that merged PRs to Claude Code more than doubled between June and September while the share of reverts fell by over 40%. Across the Anthropic monorepo, changes landing per day went from around 2,500 in May to over 30,000 in September.

It seems that now Claude writes and reviews a large % of their PRs! They have a step where Claude verifies its own work before opening a PR, then several review agents hunt for bugs in parallel and only post a finding when enough of them agree. Humans approve by risk tier, from no approval for docs and copy changes up to two reviewers for the highest risk tier.

An agent's approval never counts as a human one, and they are honest about what is missing as there are no circuit breakers yet that tighten approval when reverts go up.

It is interesting to see review being designed around the risk of each change, and it pairs nicely with the last article in this issue.

Meta on the GEM Ads Model

Meta just published a new Generative Ads Foundation model, which replaces per-surface models with one central model across Facebook, Instagram and Reels - it is fascinating to see FMs now arriving to marketing:

Meta reports an 18.9% improvement in conversions and 12% in ad click-through, with 2.5x the model complexity of the previous models. They also observe an LLM-like scaling law, as prediction quality follows a power law in sequence length and larger models benefit more from longer sequences.

The model is too expensive to serve in real time, so they transfer/distill its knowledge to hundreds of smaller vertical models. They also precompute user-ad embeddings in the background with a system called Solaris, borrowing the idea from speculative decoding.

This is pretty exciting to see for both ML and marketing practitioners, as it seems that we're now going to start seeing the ImageNet moment in many many more fields.

Will Larson on Small-Change PRs

What do you do when your engineers merge 4x more PRs than 2 years ago? Will Larson shared how Imprint introduced a framework that acceleraed PR approved with help from agents:

They claim that they were able to achieve the PR throughput whilst decreasing incidents 42% half-over-half, but median review time went up from around 33 minutes to 77.

Basically on their framework a PR qualifies when it is under 1,000 lines, has no high priority findings from their AI review bot, comes from an author with 10+ merged PRs in each service it touches and avoids shared libraries. C

It is early days for them it seems with only around 10% of changes qualifying, and Will admits the knobs will need tuning "for a long time. Maybe forever..." - which I agree!

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin