Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

How OpenAI Builds Software Now

Did you ever wonder how OpenAI builds software today internally? If you are interested on the emerging trends of "Software Factories" this is a great breakdown:

Gergely Orosz shares insights from various interviews with OpenAI engineers to showcase how they run their stack as a nine stage loop, where a human defines the outcome and Codex does pretty much everything after that.

Codex pulls context from Git, GitHub, Slack, Notion, Databricks and Datadog, writes the change and carries the PR to green. Several domain-specialist agents then review it through different lenses, before a risk classifier decides whether it needs a stricter path or can just auto-approve.

The bit I found most interesting is Perf Factory, which closes the loop from the other end; it sifts through alerts and dashboards, de-duplicates the signals, root-causes the real latency regressions and sends agents back to propose fixes.

They also talk about their "Sevbot" for incident response, which still can't take mitigation actions on its own, so engineers are very much still on call.

The adoption numbers are also pretty interesting: non-engineering teams went from roughly 0% to 90% Codex usage in four months, and PR volume put a 10x load surge through their CI systems in six - whether this is good quality or not, well, we'll see.

Netflix's Multimodal Asset Personalization at Massive Scale

Netflix's Multimodal Asset Personalization at Massive Scale; or how they use AI to pick the artwork that really gets us to click on the new titles:

Netflix uses hyper-personalisation across their stack, including the artwork and video previews you see while browsing. One topic that is particularly interesting is how they got around the cold start problem.

Their older models treated every asset as an opaque ID, so a title that had just launched had no history at all, and they fell back on popularity heuristics that ignore your taste.

Their fix was to let the model look at the picture, and concatenating a CLIP image embedding with the asset's learned ID embedding.

The preference now lives in embedding space and transfers across titles, so if you keep engaging with a "comedian's artwork", the model can favour the asset that puts them front and centre on a title it has never shown you.

Because CLIP embeddings barely move under cropping and resizing, they also collapsed five per-canvas models into one that pools signal across all of them.

The ablation is the best part of the post: image embeddings on their own were flat, the unified model on its own was flat, and only the two together won a statistically significant lift over a four week A/B test. They report the short-panel gain at 5%.

There is quite a lot of good practice buried in here too, from inverse propensity scoring on a dedicated exploration slice through to an embedding store that serves the exact same vectors at training and at inference; for production ML practitioners that last one is probably the most reusable bit here.

A 256M Model Beats 2.45B

Forecasting foundation models are taking over, with a new top model from Europe this week 🇪🇺🚀🇫🇷 The Forecasting Company released an Open 256 million parameter time series foundation model and it's going for the top 3 best in performance.

Their model t0-beta is now third on GIFT-Eval and third on fev-bench, which sounds modest until you notice they beat Datadog's Toto-2.0 on GIFT-Eval with roughly a tenth of the parameters.

Google's TimesFM-3.0 still sits at the top of both boards, however if you take performance per parameter, t0 is really competitive, and this is becoming clear seeing some of the crazy experiments they are running with WASM in the browser.

It is great to see the time series space getting this competitive. Looking forward to t0-prod! Or will it gamma?

TypeSafe Ships a System One Model

We all know that getting structured outputs from LLMs is a pain... Ex-OpenAI / RLHF co-inventor Diogo Almeida says "hold my beer" and tackles this with Jev:

Building wrappers around heavy LLMs to get them to cooperate is often hard, expensive and painful. So why not build a model trained with structured output instructions? And so they did... and released Jev.

Basically with Jev model you hand it a situation and a list of questions with their allowed answers, and it hands back one answer each with a probability attached; because the output is matched against the schema by construction it can't return a type error at all.

They trained it with something they call Reinforcement Learning for Calibrated Decisions, and they sample all the outputs in parallel instead of token by token. They claim that gets them 70ms to 500ms end to end against 3 to 329 seconds for the LLM baseline, and they list it at $0.042 per million input tokens with output free, though that number sits in a launch table and not a price list.

Output free!! I'll believe it when I see it...

The launch demo is a Doom bot, which is pretty mind blowing. Making 10 decisions a second off structured game state for about $7 an hour, and to their credit they say outright a non-AI Doom bot would play better.

AINews counted six clones in the first two days, which tells you what a nerve this one hit!

NVIDIA on Self-Optimising Agent Harnesses

It's time to build our own agentic HARNESS! NVIDIA published a paper that goes after the agent harness instead of the model:

Basically the framewokr from NVIDIA is SoL-Pi, and runs an auto-research loop that keeps proposing and testing harness optimisations. At the end the four that survived are Action Fusion, Online Context Compact, ObservationPack and an Evidence-Preserving Reducer.

NVIDIA reports cutting recorded token traffic by somewhere between 44.7% and 49%, which works out at about a third off the API bill across an hour against running the native Codex and Claude Code harnesses, while holding performance roughly at parity.

It is worth flagging that the evaluation is only 51 tasks on EdgeBench; these are clearly still early numbers, but seems worth enough publishing.

Still, the harness layer is where quite a lot of agent cost actually sits nowadays, and it is interesting to see it treated as something you can just search over automatically.

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin