Thank you for being part of over 70,000+ ML professionals and enthusiasts who receive weekly articles & tutorials on Machine Learning & MLOps 🤖 You can join the newsletter /newsletter/ ⭐

If you like the content please support the newsletter by sharing with your friends via ✉️ Email, 🐦 Twitter, 💼 Linkedin and 📕 Facebook!

This week in ML Engineering:

Raschka on Claude's Watermarking

Did you know your Claude output may have a watermark soon? Who needs a watermark when you have annoying em dashes tho?! But in all seriousness, Anthropic's approach is interesting: Raschka did a pretty interesting deep dive where he showed the watermark is applied at the token sampling stage rather than through model retraining. It is basically using a secret key together with the previous-token context to seed a deterministic "tournament sampling" process, where candidate tokens compete in pairwise rounds under random watermarking functions that assign them bit signatures. Detection is computationally cheap as it only requires the secret key and the watermarking functions, without re-running the LLM at all. It is interesting that only Anthropic holds the key, which means end users cannot independently check whether a text is watermarked, and Raschka points out the scheme can still be circumvented by passing the output through another local model for rewriting or editing. This is a great explainer on a topic we will certainly be hearing quite a lot more about; and in all seriousness the joke on em-dashes may be less of a joke, AI-slop writing-style may indeed be the watermark!

The Netflix GenRec Paper

Netflix has published the paper behind their LLM-native recommendation ranker, which is driving offline lift of about +1.6% relative MRR over their production ranker with roughly 40x less training data: The paper details is quite interesting as they use an infrequently updated Netflix-adapted foundational LLM, followed by frequent ranking-specific post-training with a reward-weighted ranking loss. It also covers the serving design, which runs prefill-only inference on vLLM to score the full candidate set in a single forward pass, with context compression from ~5,000 down to ~1,700 tokens at roughly a third of the serving cost. Netlflix shares that they ran a four-week A/B test on ~10% of traffic showing statistically significant gains on both short-term and long-term metrics. They are also candid on some of the failurse they've observed, such as over-recommending globally popular content, hallucinating out-of-catalog titles, and ignoring nuanced business constraints. Definitely worth checking out if you are building anything RecSys-adjacent!

Speculative Decoding on AMD GPUs

The vLLM team published a super detailed benchmark study of speculative decoding on AMD GPUs - a lot of great learnings for practitioners in this space: They evaluated five drafting methods (Native MTP, Gemma 4 MTP, EAGLE-3, DFlash and DSpark) which they compared across model families on MI300X hardware (288 GB VRAM each 🤯). They also had throughput multipliers from 1.27x to 2.87x on various workloads and evaluations like HumanEval. The parallel DFlash approach was frequently the best performer at longer proposal lengths, whilst per-position acceptance rates decline at later draft positions for every method. It seems the optimal proposal length was not constant across models and datasets either. For production ML practitioners the takeaway is that speculative decoding is not a free lunch you configure once and the right drafting method depends on your model, workload and hardware. It is great to see this level of empirical depth on non-NVIDIA inference!

DeepSeek's Agent Harness

DeepSeek released their new Harness, and it's one of the fastest growing OSS projects ever - record breaks every week! It seems it's taking a pi approach, where every capability is a plugin that can be swapped or recomposed - this covers models, tools, skills, sessions, sandboxes, storage, loops, scheduling and even the UI. It uses the Cordis kernel for managing dependencies and inter-plugin communication. For traceability, all agent activity lands in an append-only session log, and the Trajectory view lets you inspect system prompts, reasoning steps and tool calls. It is an interesting trend on how telemetry trasparency is being explored as harnesses evolve. They shipped it with four runtime modes, including a full standard toolset with subagents, but also a super minimal bash-and-editor setup for benchmarking. It is clear it's early days, and so far it seems like a bit of everything as opposed to something net-new, but we'll really see how it fares out in practice. Everyone is releasign a harness... it seems it's time to start working on one...

Postgres for Everything

"Just use Postgres"... this argument only continues to grow stronger throughout the years! Nowdays postgres can replace a full-text search via tsvector, NOSQL via JSONB with GIN indexing, Kafka-style queues with SELECT FOR UPDATE SKIP LOCKED, time-series with TimescaleDB, vectors with pgvector, Redis-style caching with unlogged tables, and even graphs through Apache AGE. A few weeks back I was able to take this for a spin replacing redis for postgres unlogged tables and it was quit smooth (and really nice to standardise infra), so definitely less is more here. There are no benchmarks in the post, and there arespecialised systems when Postgres no longer performs at the same level required, however the value prop of fewer moving parts is definitely a big benefit. A fun and practical read to close the week, and a good reminder that the simplest architecture that works is usually the right one.

Upcoming MLOps Events

The MLOps ecosystem continues to grow at break-neck speeds, making it ever harder for us as practitioners to stay up to date with relevant developments. A fantsatic way to keep on-top of relevant resources is through the great community and events that the MLOps and Production ML ecosystem offers. This is the reason why we have started curating a list of upcoming events in the space, which are outlined below.

You can also find our upcoming events and past talk recordings on the talks and events page.

Events we are speaking at this year:

Other relevant events:

In case you missed our talks, check our recordings below:

Open Source MLOps Tools

Check out the fast-growing ecosystem of production ML tools & frameworks at the github repository which has reached over 20,000 ⭐ github stars. We are currently looking for more libraries to add - if you know of any that are not listed, please let us know or feel free to add a PR. Here's a few featured open source libraries that we maintain:

  • SARC - Provides wrappers for popular agentic frameworks to enable guardrails and constraints that are enforced through the flow.
  • KAOS - K8s Agent Orchestration Service for managing the KAOS in large-scale distributed agentic systems.
  • Kompute - Blazing fast, lightweight and mobile phone-enabled GPU compute framework optimized for advanced data processing usecases.
  • Production ML Tools - A curated list of tools to deploy, monitor and optimize machine learning systems at scale.
  • AI Policy List - A mature list that maps the ecosystem of artificial intelligence guidelines, principles, codes of ethics, standards, regulation and beyond.
  • Agentic Systems Tools - A new list that aims to map the emerging ecosystem of agentic systems with tools and frameworks for scaling this domain

Please do support some of our open source projects by sharing, contributing or adding a star ⭐

About us

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

Check out our website

✉️ Email, 🐦 Twitter, 💼 Linkedin