Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

Raschka on Looped Transformers

Raschka is out with another mega write-up on transformer architectures, this time on GPT-6 Astra's looped transformer:

Basically, a looped transformer reuses the same blocks instead of stacking distinct ones, and the Astra model runs its 22 blocks twice and gets 44 effective block applications while storing one set of weights.

It seems that this really helps save parameter efficiency, however the forward and backward pass still cost what 44 distinct blocks would, and the KV cache needs separate entries per pass, so nothing is saved there either.

The team apparently found two loops optimal, with more giving minimal gains and destabilising training; the breakdown also includes a breakdown of Universal Transformers through Mixture-of-Recursions and SMELT.

The part I found most useful is his push-back on the worry that looping actually can hide the reasoning traces...

OpenAI has been hiding traces since o1 regardless of architecture, and Raschka reckons Astra's shorter traces just reflect a model that makes fewer mistakes and backtracks less; his analogy is that stronger students need less scratch paper.

Worth keeping in mind that all of this is inferred from public statements, as OpenAI has not confirmed a looped design at all.

OpenAI Scaling Storage to 1B Users

OpenAI published a 2-part series on scaling to 70 million requests a second over 500 petabytes:

The scale that OpenAI has to deal with is impressive... not almost 40 regions and more than a billion people a week. This is also with 10x year-over-year growth for three years running.

It seems their initiative started in 2023 as a small Python library on a single Cosmos DB, and is now the second largest service at OpenAI by core count.

I found intresting how OpenAI emphasises consciously staying using Python to continue momentum, and unlock cost by migrating later. In Q2 2026 two engineers with Codex and GPT-5.5 rewrote the service in Rust, which now serves 95% of production at 6x the CPU efficiency and 15x the memory efficiency.

The debugging stories are also pretty interesting; their pods kept locking up under load, and Python's aiohttp connector turned out to default to LIFO connection reuse, so the slow overloaded servers returned their connections last and kept getting picked again (then switched to FIFO).

They also make a good case for deliberately exposing a NoSQL API as cheap SQL that is expensive to run used to take out their Postgres regularly. Interesting to see what part two says about the storage layer.

Pinterest on Billion-Scale Embedding Retrieval

Pinterest shared how they are evolving their Billion-Scale Embedding Retrieval models, and there's quite a few great lessons:

First, Quantization is already rolled out everywhere, and they benchmarked scalar against product quantization on 100 million GraphSage embeddings.

For Pinterest, Product Quantization cuts the HNSW index by 74% and the IVF index by 93% but drops recall to 70-80%, while Scalar Quantization cuts them by 59% and 75% and holds recall above 90%.

They chose per-use-case with online A/B tests and report 20-30% savings on serving.

The SSD work was also particularly interesting, as they benchmarked DiskANN against SPANN and went with SPANN, with one tweak on the paper, as they quantize the on-disk embedding store with PQ but keep the centroids at full precision.

It seems that this gets them 3x the QPS of DiskANN at a third of the latency for a 5% recall drop; on a 5 billion embedding index they saved over 40% of the CPU time against HNSW.

Finally it seems they are moving to late interaction with ColBERT-style scoring next, though that is still a pilot. + great to see numbers this specific on the cost side.

Google Inverts Tool-Use Data Generation

Google Research published ToolGrad, a new way to build the datasets optimize tool-use for foundation models, and it's intersting how it inverts the usual approach:

A "tool-use" dataset is basically a user prompt paired with the chain of API calls that answers that prompt.

The usual way to build one is to invent a plausible prompt first and then set search agents loose to find a chain that satisfies it - this is often suboptimal.

ToolGrad basically invers this approach, as they build the API chain first and actually execute every call, then write the user prompt that would have produced it.

It seems this makes the label correct by construction, since the chain demonstrably ran before the prompt existed.

Google reports a 99.8% pass rate against the query-first baseline across ToolBench's 16k APIs, and a Gemma-3 12B fine-tuned on the generated set landing 83.1 on BFCL against Gemini 2.5-Pro's 83.2.

Another thing that for me is quite relevant is how to think about these datasets and approaches as potentially relevant for evals in the application layer.

Cognition's SWE-2 Pushes the Pareto Frontier

Cognition have shipped their new SWE-2, and it seems new records are set every week... they are just behind Fable 5.1 but at 64% less cost

It's basically post-trained from Kimi K3 with an RL setup that trains every reasoning-effort level in one run so the whole cost curve moves rather than a single point on it.

The cost penalty is derived rather than tuned, which I thought was interesting: they prove it has to be linear, then set its slope from the base model's own Pareto curve.

Important to note that this is their own bench. They also report Kimi K3 passing 54.5% on a Simplified Chinese censorship eval against SWE-2's 95.2%, which is quite a jump for a post-train.

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin