Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

GPT-6 Astra Breaks the Benchmarks

GPT-6 Astra is out, and at this point the benchmark tables are getting quite ridiculous!

OpenAI claims 99.9% on ARC-AGI-3 and a clean 100% on ExploitBench, which sounds like either Astra hacked HF again to get the results or we need new benchmarks...

It is interesting to see that it's still not unanimous though, as it seems to fall behind Fable on Humanity's Last Exam, and behind also Opus on the Artificial Analysis indices.

It is interesting to see the pareto curve of cost vs task complexity, I am all for this moving on the cheaper side... main question is how subsidised it is - and for how long!

It's interesting they also claim Astra's written reasoning is harder to monitor than Sol's - it is good to see a lab publishing the eval where their own model got worse (albeit likely also a lot of marketing around).

Netflix on the Judge Lifecycle

Netflix published their approach to building and monitoring LLM judges at scale. Their main challenge is - wait for it - data drift:

Basically Netflix realised that if the judge starts approving bad output, nothing downstream will flag it, so this is a critical barrier to ensure is bulletproof.

Their main use-case for LLM Judge is recommendation explanations - basically telling you why a title was suggested. They generate this in the hundreds of thousands per week across their rolling catalog.

This is pretty cool as it gooes beyond standard evals, as they propose a technique they call Reasoning-Aligned Rubric Tuning, which treats "right label, wrong reason" as its own error class.

This Meta-Judge compares the judge's written rationale against a human rater's, and a Reflector then edits the rubric text rather than retraining anything.

It seems they also run a weekly human review of around 300 explanations with at least three raters each, and they measure drift against the raters' own spread, so a harder week widens human disagreement and widens the band with it.

Super cool analysis, this is actually something that I had been playing around on the evals front, but it seems that now there are "pattern names" also arising.

Databricks on Agentic Kernel Generation

Should we let an agent write our GPU kernels? Databricks says yes:

Databricks have built a harness called Proteus that proposes kernels, validates them against a controlled reference, times only the survivors and iterates, and they report generated Qwen 3.5 122B kernels running 1.8x to 5.2x faster than the best implementations in vLLM.

They caught the harness cheating tho, namey candidates reusing compiled code from an earlier attempt so a rebuild looked cheaper.

They also saw other forms of reating like replaying batched launches through a CUDA graph while the baseline still launched each piece separately, and fitting the visible test sizes while falling over on unseen ones.

The fixes are still interesting, time both sides identically with more than one timer, clear leftover state, re-time winners before promoting them, and hold out tests the candidate cannot see.

It is quite cool to see reward hacking turn up as a plain engineering problem rather than an alignment talking point.

Google TimesFM-3 Goes Multivariate

Google Research has released a new time series foundation model! TimesFM-3 finally closes the gap and catches up:

Basically everything up to TimesFM-2.5 was strictly univariate, while almost every real forecasting problem has covariates hanging off it.

The model is 330M parameters trained on over a trillion time points (still blows my mind how large these training sets are), and it handles multiple targets, past covariates and known-future signals (eg like planned promotions) in a single forward pass.

They alternate causal attention across time with full attention across series, and they drop the autoregressive decode so the whole horizon comes out at once.

Google claim the top average rank among pre-trained foundation models on Gift-Eval, FEV-Bench and Time - but indeed, let's see what the community says once it's taken for a run.

Polars 2.0 Goes Streaming First

The Polars 2.0 release candidate is out 🚀 And this one is a big deal:

It seems they are goingn full in on streaming engine by default; every LazyFrame query now runs on there by default, and they say they expect it to be easily 5x faster in aggregate with large memory improvements.

It seems they have a tradeoff as part of the upgrade on breaking changes, as row order is no longer guaranteed for join + group_by + unpivot unless you ask for it with maintain_order.

The team say they hope 2.0 is a boring experience, which is exactly the right ambition :) definitely worth a try with polars==2.0rc1 - if you find any issues file an issue!

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin