Stanford Study on Local LLMs

A study from Stanford University showed that 71.3% of chatgpt queries could be accurately answered by a local model: Stanford researchers have published a large empirical study that explores whether local LLM inference can take on part of today’s cloud-served workload. The paper defines intelligence per watt as task accuracy divided by power consumption, and evaluates 20+ local language models across 8 accelerators on more than 1M single-turn chat and reasoning queries. The main result is that local models can correctly handle 88.7% of the studied queries when routed to the best local model, although coverage is much stronger for chat and knowledge-style tasks than for harder technical reasoning. The longitudinal results are also relevant for ML platform teams as it seems 2023-2025 local-query coverage increased from 23.2% to 71.3%, while intelligence per watt improved by 5.3x. Cloud accelerators still retain a clear per-query efficiency advantage on identical models, but hybrid local-cloud routing changes the system-level tradeoff, as just with an 80%-accurate router, the simulated deployment reduces energy by 64.3%, compute by 61.8%, and cost by 59.0% against a batched cloud baseline. This is super promising, as it shows that there's so much potential in local models that is really untapped - there's really a lot to come from this space.

Snorkel's Senior SWE-Bench

SnorkelAI has dropped a new SWE-Bench for LLMs that attempts to capture Senior Engineering type tasks that extend the scope towards under-specified feature requests, runtime bug investigation, behavioral verification, and code-quality assessment: It is interesting to see the rise of SWE benchmarks attempting different schools of thought to bring balanced reviews of new models; I like that this one is based on real PRs, uses a mix of pre-written verifiers, an adaptive agents to ensure a useful distinction between code that passes tests and code that fits the surrounding codebase. In the reported results, GPT-5.5 has the highest basic solve rate at 55.0%, while Claude Opus 4.8 has the highest tasteful solve rate at 24.0%; GPT-5.5 is also more efficient, averaging 36.3K output tokens and 89 agent steps per task, compared with 117.1K tokens and 131 steps for Claude Opus 4.8. For production ML practitioners building or evaluating coding-agent workflows it is worth noting that pass rates are an incomplete signal now, as teams should also measure root-cause correctness, design quality, abstraction fit, and even codebase-pattern alignment.

Google DeepMind's Tabular Foundation Model

Google Research has now entered the Tabular Foundation Model space! They have released TabFM, a tabular foundation model for classification and regression, which seems to mostly use learnings from the first-mover startups in the space (but still awesome): This model basically treats each table as an in-context learning task (instead of fitting a dataset-specific model), and this is useful because TabFM instead combines row and column attention, row compression and an ICL Transformer, with pre-training on hundreds of millions of synthetic datasets generated from structural causal models. Google reports strong TabArena results across 51 classification and regression datasets, including a zero-shot setting that runs in a single forward pass and an ensemble variant with cross features, SVD features, NNLS blending and calibration. For production ML practitioners, the most immediate value is in fast baselines and prototyping clearly. However, unfortunately it seems that the public weights are non-commercial (boo!), classification is limited to 10 classes, memory scales with the number of context rows, and (obviously) domain-specific validation remains necessary before use in critical systems.

Performance Per Dollar Improving

Performance per dollar in local LLMs is getting faster and cheaper! Wafer shows how they served GLM5.2 on AMD MI355X at 2626 tok/s/node and 213 tok/s single stream at over 2x lower cost than Blackwell. This is an interesting case study that showcases how much opportunity there is on local model inference efficiency; 2.4 RPS on a 20k input / 1k output workload with a 60% cache-hit rate, and 213 tok/s single-stream on a 10k input / 1.5k output test is impressive. The result is relevant for production ML teams because the gains came mostly from framework and configuration work rather than new custom kernels: Wafer quantized bf16 GLM-5.2 to MXFP4 with AMD Quark, selected sglang after testing vLLM and ATOM, fixed ROCm-specific speculative decoding issues, enabled FP8 KV cache, and tuned MoE kernel selection for GLM’s fp4 shapes. Vercel has also made GLM 5.2 Fast via Wafer available through AI Gateway, with its own benchmarks reporting higher throughput than other serverless GLM-5.2 providers across small-context, large-context, and tool-call scenarios.

CVEs Spike After Mythos Release

Disclosure of serious cyber vulnerabilities spiked around the release of Claude Mythos Preview: is it time to rethink public vulnerability reporting/mgmt? Epoch AI reported that public disclosure of high/critical severity CVEs have been spiking right after Anthropic announced Claude Mythos Preview. In June 2026 the 21 major organizations tracked by Epoch published around 1.5k high/critical severity CVEs which is more than 3.5 times the previous monthly record before Mythos Preview was announced. Anthropic’s related Project Glasswing update claims that roughly 50 partners have found more than 10k high/critical severity vulnerabilities, while also stating that the limiting factor has shifted from discovery to verification, disclosure, patch development, and deployment. OpenAI’s Daybreak initiative describes a similar defensive workflow, focused on finding, validating, and fixing vulnerabilities before attackers can use them. It seems that the dawn of agentic cybersecurity has come, and we'll be likely seeing quite a few shifts in mindset, approach and criticality to security across every domain and industry.

The Institute for Ethical AI & Machine Learning is a European research centre that carries out world-class research into responsible machine learning.

✉️ Email, 🐦 Twitter, 💼 Linkedin