Awesome Production Machine Learning

AIDE is an open-source ML engineering agent that uses a tree search algorithm to autonomously explore, implement, and evaluate solution strategies for machine learning tasks.

Automated feature, model, and hyperparameter selection for tabular, image, and text data on top of popular machine learning libraries (Scikit-Learn, LightGBM, CatBoost, PyTorch, MXNet).

AutoML library for Keras based on "Auto-Keras: Efficient Neural Architecture Search with Network Morphism".

Framework to automate algorithm and hyperparameter tuning for sklearn.

Ax is an accessible, general-purpose platform for understanding, managing, deploying, and automating adaptive experiments.

BoTorch is a library for Bayesian Optimization built on PyTorch.

EvalML is an AutoML library which builds, optimizes, and evaluates machine learning pipelines using domain-specific objective functions.

Feature-engine is a Python library that contains several transformers to engineer features for use in machine learning models.

An open source framework for automated feature engineering.

FLAML is a fast library for automated machine learning & tuning.

Set of open-source hyperparameter optimization frameworks, including the winning submission to the NeurIPS 2020 Black-Box Optimisation Challenge tested on hyperparameter tuning tasks.

A Kubernetes-based system for Hyperparameter Tuning and Neural Architecture Search.

Keras Tuner is an easy-to-use, distributable hyperparameter optimisation framework that solves the pain points of performing a hyperparameter search. Keras Tuner makes it easy to define a search space and leverage included algorithms to find the best hyperparameter values.

Optuna is an automatic hyperparameter optimisation software framework, particularly designed for machine learning.

OSS Vizier is a Python-based service for black-box optimisation and research, one of the first hyperparameter tuning services designed to work at scale.

A gradient boosting machine that doesn't need hyperparameter optimization, with a simple budget parameter to control model complexity.

Automation of sklearn pipeline creation (including feature selection, pre-processor, etc.).

Automatic extraction of relevant features from time series.

Accelerate abstracts exactly and only the boilerplate code related to multi-GPU/TPU/mixed-precision and leaves the rest of your code unchanged.

Adapters is a unified library for parameter-efficient and modular transfer learning.

BitBLAS is a library to support mixed-precision BLAS operations on GPUs

Cache-DiT is built on top of Diffusers and supports nearly all DiTs, providing hybrid cache acceleration (DBCache, TaylorSeer, SCM, etc.) and comprehensive parallelism optimizations including Context Parallelism, Tensor Parallelism, and hybrid 2D/3D parallelism, with compatibility for compilation, CPU offloading, and quantization.

A unified deep learning system for big model era, which helps users to efficiently and quickly deploy large AI model training and inference.

Composer is a PyTorch library that enables you to train neural networks faster, at lower cost, and to higher accuracy.

Built based on the Apache Arrow columnar memory format, cuDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.

cuML is a suite of libraries that implement machine learning algorithms and mathematical primitives functions that share compatible APIs with other RAPIDS projects.

An implementation of NumPy-compatible multi-dimensional array on CUDA. CuPy consists of the core multi-dimensional array class, cupy.ndarray, and many functions on it.

A novel evolutionary computation framework for rapid prototyping and testing of ideas. It seeks to make algorithms explicit and data structures transparent. It works in perfect harmony with parallelisation mechanisms such as multiprocessing and SCOOP.

DeepEP is a communication library tailored for Mixture-of-Experts (MoE) and expert parallelism (EP). It provides high-throughput and low-latency all-to-all GPU kernels, which are also known as MoE dispatch and combine. The library also supports low-precision operations, including FP8.

DGL is an easy-to-use, high performance and scalable Python package for deep learning on graphs.

DLRover makes the distributed training of large AI models easy, stable, fast and green.

Distributed parallel processing framework for Pandas and NumPy computations.

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

FlagGems is a high-performance general operator library implemented in OpenAI Triton. It builds on a collection of backend neutral kernels that aims to accelerate LLM training and inference across diverse hardware platforms.

A fast, flexible machine learning library written entirely in C++ from the Facebook AI Research and the creators of Torch, TensorFlow, Eigen and Deep Speech.

A neural network library and ecosystem for JAX designed for flexibility.

GPUStack is an open-source GPU cluster manager for running AI models.

Decentralized deep learning in PyTorch.

Uber's distributed training framework for TensorFlow, Keras, and PyTorch.

Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more.

Blazing fast, lightweight and mobile phone-enabled Vulkan compute framework optimized for advanced GPU data processing usecases.

Lava is an open source framework to develop applications for neuromorphic hardware architectures.

Liger Kernel is a collection of Triton kernels designed specifically for LLM training.

LightGBM is a gradient boosting framework that uses tree based learning algorithms.

MLX is an array framework for machine learning on Apple silicon.

Speed up your Pandas workflows by changing a single line of code.

TensorRT is a C++ library for high-performance inference on NVIDIA GPUs and deep learning accelerators.

Nevergrad is a gradient-free optimisation platform.

Norse aims to exploit the advantages of bio-inspired neural components, which are sparse and event-driven - a fundamental difference from artificial neural networks.

A compiler for Python array and numerical functions.

Optimum is an extension of Transformers and Diffusers, providing a set of optimization tools enabling maximum efficiency to train and run models on targeted hardware while keeping things easy to use.

Parameter-Efficient Fine-Tuning (PEFT) methods enable efficient adaptation of pre-trained language models (PLMs) to various downstream applications without fine-tuning all the model's parameters.

PaddlePaddle is a framework to perform large-scale deep network training, using data sources distributed across hundreds of nodes.

PyG (PyTorch Geometric) is a library built upon PyTorch to easily write and train Graph Neural Networks (GNNs) for a wide range of applications related to structured data.

PyTorch Lightning pretrains, finetunes and deploys AI models on multiple GPUs, TPUs with zero code changes.

PyTorch is a library to develop and train neural network based deep learning models.

Ray is a flexible, high-performance distributed execution framework for machine learning.

SetFit is an efficient and prompt-free framework for few-shot fine-tuning of Sentence Transformers.

Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learning research.

A Data Streaming Library for Efficient Neural Network Training.

TensorFlow is a leading library designed for developing and deploying state-of-the-art machine learning applications.

ThunderKittens is a framework to make it easy to write fast deep learning kernels in CUDA.

TorchOpt is an efficient library for differentiable optimization built upon PyTorch.

Triton is a language and compiler for writing highly efficient custom Deep-Learning primitives. The aim of Triton is to provide an open-source environment to write fast code at higher productivity than CUDA, but also with higher flexibility than other existing DSLs.

Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular datasets. Vaex uses memory mapping, zero memory copy policy and lazy computations for best performance (no memory wasted).

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

XGBoost is an optimized distributed gradient boosting library designed to be highly efficient, flexible and portable.

YDF (Yggdrasil Decision Forests) is a library to train, evaluate, interpret, and serve Random Forest, Gradient Boosted Decision Trees, CART and Isolation forest models.

Bitsandbytes library is a lightweight Python wrapper around CUDA custom functions, in particular 8-bit optimizers, matrix multiplication (LLM.int8()), and 8 & 4-bit quantization functions.

Flexible and powerful tensor operations for readable and reliable code.

Scikit-learn is a powerful machine learning library that provides a wide variety of modules for data access, data preparation and statistical model building.

snnTorch is a deep and online learning library with spiking neural networks.

torchdistill offers various state-of-the-art knowledge distillation methods and enables you to design (new) experiments simply by editing a declarative yaml config file instead of Python code.

The torchkeras library is a simple tool for training neural network in pytorch jusk in a keras style.

veScale is a PyTorch native LLM training framework.

yellowbrick is a matplotlib-based model evaluation plots for scikit-learn and other machine learning libraries.

Argilla helps domain experts and data teams to build better NLP datasets in less time.

Python library for data-centric AI. Can automatically: find mislabeled data, detect outliers, estimate consensus + annotator-quality for multi-annotator datasets, suggest which data is best to (re)label next.

Web-based image segmentation tool for object detection, localization and keypoints

CVAT (Computer Vision Annotation Tool) is OpenCV's web-based annotation tool for both videos and images for computer algorithms.

Open source text annotation tools for humans, providing functionality for sentiment analysis, named entity recognition, and machine translation.

Gretel Synthetics is a synthetic data generators for structured and unstructured text, featuring differentially private learning.

Multi-domain data labeling and annotation tool with standardized output format.

An open source tool to curate, annotate, and manage vision datasets (images and videos). Supports embedding-based auto-selection, annotation, and auto-labeling for bounding boxes and segmentation.

NeMo Curator is a GPU-accelerated framework for efficient large language model data curation.

The data scientist's open-source choice to scale, assess and maintain natural language data.

Synthetic Data Vault (SDV) is a Synthetic Data Generation ecosystem of libraries that allows users to easily learn single-table, multi-table and timeseries datasets to later on generate new Synthetic Data that has the same format and statistical properties as the original dataset.

Hitachi's Open source tool for labelling camera and LIDAR data.

synthcity is a library for generating and evaluating synthetic tabular data.

Synthetic tabular data generation using GANs (CTGAN), Diffusion Models, and LLMs with adversarial filtering, privacy metrics, and sklearn integration.

ViPE is a spatial AI tool for annotating camera poses and dense depth maps from raw videos.

YData Synthetic is a package to generate synthetic tabular and time-series data leveraging the state of the art generative models.

Data Pipeline framework built in Python, including scheduler, DAG definition and a UI for visualisation.

Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic.

Workflow scheduler for Hadoop jobs.

Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflows is implemented as a Kubernetes CRD (Custom Resource Definition).

Unified interface for constructing and managing machine learning workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

DataTrove is a library to process, filter and deduplicate text data at a very large scale.

A data orchestrator for machine learning, analytics, and ETL.

ETL tool for running transformations inside data warehouses.

Lyft's Cloud Native Machine Learning and Data Processing Platform - (Demo).

Job orchestration engine to interface and trigger the execution of jobs from Hadoop-based systems.

Hamilton is a micro-orchestration framework for defining dataflows. Runs anywhere python runs (e.g. jupyter, fastAPI, spark, ray, dask). Brings software engineering best practices without you knowing it. Use it to define feature engineering transforms, end-to-end model pipelines, and LLM workflows. It complements macro-orchestration systems (e.g. kedro, luigi, airflow, dbt, etc.) as it replaces the code within those macro tasks. Comes with a self-hostable UI that captures lineage & provenance, execution telemetry & data summaries, and builds a self-populating catalog; usable in development as well as production.

Instill VDP (Versatile Data Pipeline) aims to streamline the data processing pipelines from inception to completion.

Instructor makes it easy to get structured data like JSON from LLMs like GPT-3.5, GPT-4, GPT-4-Vision, and open-source models.

Kedro is a workflow development tool that helps you build data pipelines that are robust, scalable, deployable, reproducible and versioned.

Luigi is a Python module that helps you build complex pipelines of batch jobs, handling dependency resolution, workflow management, visualisation, etc..

A framework for data scientists to easily build and manage real-life data science projects.

Open source distributed processing framework build on Kubernetes focused mainly on dynamic building of production machine learning pipelines - (Video).

The fastest way to build data pipelines. Develop iteratively, deploy anywhere.

Open-source Python library providing declarative, incremental data infrastructure for building and managing multimodal AI workloads.

Workflow management system that makes it easy to take your data pipelines and add semantics like retries, logging, dynamic mapping, caching, failure notifications, and more.

SeqIO is a library for processing sequential data to be fed into downstream sequence models.

Workflow management system for reproducible and scalable data analyses.

General-purpose machine learning pipeline for generating embedding vectors using one or many ML models.

unstructured streamlines and optimizes the data processing workflow for LLMs, ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and many more.

ZenML is an extensible, open-source MLOps framework to create reproducible ML pipelines with a focus on automated metadata tracking, caching, and many integrations to other tools.

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Deepnote is a drop-in replacement for Jupyter with an AI-first design, sleek UI, new blocks, and native data integrations. Use Python, R, and SQL locally in your favorite IDE, then scale to Deepnote cloud for real-time collaboration, Deepnote agent, and deployable data apps.

Web interface python sandbox environments for reproducible development

Reactive Python notebook — run reproducible experiments, execute as a script, deploy as an app, and version with git.

Papermill is a library for parameterizing notebooks and executing them like Python scripts.

Polynote is an experimental polyglot notebook environment. Currently, it supports Scala and Python (with or without Spark), SQL, and Vega.

The rmarkdown package is a next generation implementation of R Markdown based on Pandoc.

Stencila is a platform for creating, collaborating on, and sharing data driven content. Content that is transparent and reproducible.

Voilà turns Jupyter notebooks into standalone web applications that can e.g. be used as dashboards.

.NET Interactive takes the power of .NET and embeds it into your interactive experiences.

AIStore is a lightweight object storage system with the capability to linearly scale out with each added storage node and a special focus on petascale deep learning.

A virtual distributed storage system that bridges the gab between computation frameworks and storage systems.

In-memory columnar representation of data compatible with Pandas, Hadoop-based systems, etc..

A high performance real-time analytics database. Check this article for introduction.

Hudi is a transactional data lake platform that brings core warehouse and database functionality directly to a data lake. Hudi is great for streaming workloads, and also allows creation of efficient incremental batch pipelines. Supports popular query engines including Spark, Flink, Presto, Trino, Hive, etc. More info here.

Iceberg is an ACID-compliant, high-performance format built for huge analytic tables (containing tens of petabytes of data), and it brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time. More info here.

A memory-centric distributed database, caching, and processing platform for transactional, analytical, and streaming workloads delivering in-memory speeds at petabyte scale - Demo.

On-disk columnar representation of data compatible with Pandas, Hadoop-based systems, etc..

A realtime distributed OLAP datastore. Comparison of the open source OLAP systems for big data: ClickHouse, Druid, and Pinot is found here.

Casibase is a LangChain-like RAG (Retrieval-Augmented Generation) knowledge database with web UI and Enterprise SSO.

Chroma is an open-source embedding database.

ClickHouse is an open source column oriented database management system.

Delta Lake is a storage layer that brings scalable, ACID transactions to Apache Spark and other big-data engines.

Gel supercharges Postgres with a modern data model, graph queries, Auth & AI solutions, and much more.

GPTCache is a library for creating semantic cache for large language model queries.

Scalable datastore for metrics, events, and real-time analytics.

Milvus is a cloud-native, open-source vector database built to manage embedding vectors generated by machine learning models and neural networks.

Marqo is an end-to-end vector search engine.

pgvector helps with vector similarity search for Postgres.

PostgresML is a machine learning extension for PostgreSQL that enables you to perform training and inference on text and tabular data using SQL queries.

Redis is an open-source, in-memory data store that supports vector similarity search, making it suitable for AI/ML applications such as semantic search and recommendation systems.

Simple, safe way to store and distribute tensors.

An open-source time-series SQL database optimized for fast ingest and complex queries packaged as a PostgreSQL extension - (Video).

A low-latency vector search engine (GraphQL, RESTful) with out-of-the-box support for different media types. Modules include Semantic Search, Q&A, Classification, Customizable Models (PyTorch/TensorFlow/Keras), and more.

Python implementation of chunked, compressed, N-dimensional arrays designed for use in parallel computing.

Apache Beam is a unified programming model for Batch and Streaming.

Open source stream processing framework with powerful stream and batch processing capabilities.

Kafka client library for building applications and microservices where the input and output are stored in kafka clusters.

Distributed stream processing framework. It uses Apache Kafka for messaging, and Apache Hadoop YARN to provide fault tolerance, processor isolation, security, and resource management.

Micro-batch processing for streams using the apache spark framework as a backend supporting stateful exactly-once semantics.

Flexible Python-centric stateful stream processing framework built on top of Rust engine.

A modern broker-agnostic streaming Python framework supporting Apache Kafka, RabbitMQ and NATS protocols, inspired by FastAPI and easily integratable with other web frameworks.

MOA (Massive Online Analysis) is an open source framework for Big Data stream mining.

Fast, deterministic streaming of large datasets from cloud storage for distributed model training.

A distributed SQL streaming database that unifies stream processing and low-latency serving, ideal for building and serving features for online machine learning.

Library for reading and writing large multi-dimensional arrays.

Agenta provides end-to-end tools for the entire LLMOps workflow: building (LLM playground, evaluation), deploying (prompt and configuration management), and (LLM observability and tracing).

AirLLM optimizes inference memory usage, allowing 70B large language models to run inference on a single 4GB GPU card without quantization, distillation and pruning.

AITemplate (AIT) is a Python framework that transforms deep neural networks into CUDA (NVIDIA GPU) / HIP (AMD GPU) C++ code for lightning-fast inference serving.

BentoML is an open source framework for high performance ML model serving.

BISHENG is an open LLM application devops platform, focusing on enterprise scenarios.

Machine Learning production server for TensorFlow, XGBoost and Cafe models written in C++ and maintained by Jolibrain.

NVIDIA Dynamo is a high-throughput, low-latency inference framework designed for serving generative AI and reasoning models in multi-node distributed environments.

exo helps you run your AI cluster at home with everyday devices.

Genkit is an open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations.

GoModel is a self-hosted AI gateway written in Go that exposes a unified OpenAI-compatible API across OpenAI, Anthropic, Gemini, Groq, xAI, Ollama and other providers, with routing, usage tracking, rate limits, and guardrails.

A fast, production-ready inference server for computer vision supporting deployment of many popular model architectures and fine-tuned models. With Inference, you can deploy models such as YOLOv5, YOLOv8, CLIP, SAM, and CogVLM on your own hardware using Docker.

Infinity is a high-throughput, low-latency REST API for serving text-embeddings, reranking models and clip.

IPEX-LLM is a PyTorch library for running LLM on Intel CPU and GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) with very low latency.

LiteLLM is a Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq.

LiteRT (formerly TensorFlow Lite) is Google's high-performance runtime for on-device AI inference, enabling deployment of machine learning models on mobile, embedded, and edge devices.

LiteRT-LM is Google's production-ready, high-performance inference framework for deploying Large Language Models on edge devices, with cross-platform support for Android, iOS, Web, Desktop, and IoT.

LitServe is a flexible serving engine for AI models built on FastAPI. It supports custom inference engines for models, agents, multi-modal systems, RAG, and complex ML pipelines.

Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets.

Kiln is an OSS tool for fine-tuning LLM models, synthetic data generation, and collaborating on datasets.

KServe provides a Kubernetes Custom Resource Definition for serving predictive and generative ML.

KTransformers is a flexible framework for experiencing cutting-edge LLM inference optimizations.

Langtrace is an open-source, Open Telemetry based end-to-end observability tool for LLM applications, providing real-time tracing, evaluations and metrics for popular LLMs, LLM frameworks, vectorDBs and more.

LeptonAI Python library allows you to build an AI service from Python code with ease.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

llama.cpp is an open source software library that performs inference on various large language models such as Llama.

A terminal tool that right-sizes LLM models to your system's RAM, CPU, and GPU. Detects your hardware, scores each model across quality, speed, fit, and context dimensions, and tells you which ones will actually run well on your machine.

LMCache is a high-performance KV cache layer that accelerates LLM inference.

LMDeploy is a toolkit for compressing, deploying, and serving LLM.

LM Studio is a tool for deploying LLM models locally on the computer, even on a relatively modest machine, provided it meets the minimum requirements.

LocalAI is a drop-in replacement REST API that's compatible with OpenAI API specifications for local inferencing.

MindsDB is the platform to create, serve, and fine-tune models in real-time from your database, vector store, and application data.

mini-sglang is a lightweight and efficient serving framework for large language models.

MLRun is an open MLOps framework for quickly building and managing continuous ML and generative AI applications across their lifecycle.

An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more.

Docker Model Runner makes it easy to manage, run, and serve AI models using Docker, supporting LLMs and other AI models pulled directly from Docker Hub or any OCI-compliant registry.

A rust-powered and multi-stage pipelined model server which offers dynamic batching and more. Super easy to implement and deploy as micro-services.

nano-vllm is a lightweight vLLM implementation built from scratch, providing fast offline inference with optimization techniques such as prefix caching, tensor parallelism, and CUDA graph.

An Easy-to-Use and High-Performance AI deployment framework.

A high-performance "serverless" framework focused on data, I/O, and compute-intensive workloads. It is well integrated with popular data science tools, such as Jupyter and Kubeflow; supports a variety of data and streaming sources; and supports execution over CPUs and GPUs.

OpenLLM allows developers to run any open-source LLMs (Llama 3.1, Qwen2, Phi3 and more) or custom models as OpenAI-compatible APIs with a single command.

OpenVINO is an open-source toolkit for optimizing and deploying AI inference.

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with built-in inference engine for RAG, making it a powerful AI deployment solution.

OptiLLM is an OpenAI API-compatible optimizing inference proxy that implements 20+ state-of-the-art techniques to dramatically improve LLM accuracy and performance on reasoning tasks - without requiring any model training or fine-tuning.

PowerInfer is a CPU/GPU LLM inference engine leveraging activation locality for your device.

Prompt2Model is a system that takes a natural language task description (like the prompts used for LLMs such as ChatGPT) to train a small special-purpose model that is conducive for deployment.

RamaLama is an open-source tool that simplifies the local use and serving of AI models for inference through OCI containers, eliminating the need to configure the host system.

RunAnywhere is a production-ready SDK for running AI models (LLMs, speech-to-text, text-to-speech) on-device for iOS, Android, React Native, and Flutter - enabling private, offline, and fast mobile AI applications.

Open source platform for deploying and machine learning models in Kubernetes - (Video).

SGLang is a fast serving framework for large language models and vision language models.

Open-source inference server and production cluster for embeddings, reranking, and extraction. 85+ pre-configured models across dense, sparse, multi-vector, vision, rerankers, and extractors. Ships with Helm, KEDA autoscaling, Grafana dashboards, and Terraform.

SkyPilot is a framework for running LLMs, AI, and batch jobs on any cloud, offering maximum cost savings, highest GPU availability, and managed execution.

High-performant framework to serve Tensorflow models via grpc protocol able to handle 100k requests per second per core.

Large Language Model Text Generation Inference.

TorchServe is a flexible and easy to use tool for serving PyTorch models.

torchtune is a PyTorch library for easily authoring, post-training, and experimenting with LLMs.

Transformer Lab is an open-source LLM workspace for finetuning, evaluating, exporting, and testing models locally across inference engines and platforms.

Triton is a high performance open source serving software to deploy AI models from any framework on GPU & CPU while maximizing utilization.

Vercel AI is a TypeScript toolkit designed to help you build AI-powered applications using popular frameworks like Next.js, React, Svelte, Vue and runtimes like Node.js.

Search, make inferences in and organize vectors, tensors, text and structured data, at serving time and any scale.

vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.

AlpacaEval is an automatic evaluator for instruction-following language models.

ANN-Benchmarks is a benchmarking environment for approximate nearest neighbor algorithms search.

ARES is a framework for automatically evaluating Retrieval-Augmented Generation (RAG) models.

BEIR is a heterogeneous benchmark containing diverse IR tasks. It also provides a common and easy framework for evaluation of your NLP-based retrieval models within the benchmark.

Code Generation LM Evaluation Harness is a framework for the evaluation of code generation models.

COMET is an open-source framework for machine learning evaluation.

C-Eval is a comprehensive Chinese evaluation suite for foundation models.

Deepchecks is a holistic open-source solution for all of your AI & ML validation needs, enabling you to test your data and models from research to production thoroughly.

DeepEval is a simple-to-use, open-source evaluation framework for LLM applications.

DomainBed is a test suite containing benchmark datasets and algorithms for domain generalization

EvalAI is an open-source platform for evaluating and comparing AI algorithms at scale.

Evalchemy is a unified and easy-to-use toolkit for evaluating post-trained language models.

EvalPlus is a robust evaluation framework for LLM4Code, featuring expanded HumanEval+ and MBPP+ benchmarks, efficiency assessment (EvalPerf), and a secure, extensible evaluation toolkit.

Evals is a framework for evaluating OpenAI models and an open-source registry of benchmarks.

EvalScope is a streamlined and customizable framework for efficient large model evaluation and performance benchmarking.

Evaluate is a library that makes evaluating and comparing models and reporting their performance easier and more standardized.

Evidently is an open-source framework to evaluate, test and monitor ML and LLM-powered systems.

Open-source self-hostable end-to-end agent engineering and optimization platform unifying tracing, evals, simulations, datasets, gateway, and guardrails for LLM and AI agent applications.

GAOKAO-Bench is an evaluation framework that uses Chinese National College Entrance Examination (GAOKAO) questions as a dataset to assess large models' language comprehension and logical reasoning abilities.

Giskard is an open-source Python library that automatically detects performance, bias & security issues in AI applications.

guidellm is a benchmarking and performance evaluation tool for large language model inference systems.

Harbor is a framework for evaluating and optimizing agents and language models, supporting parallel experiments across container environments with built-in benchmark and environment management.

HumanEval is a benchmark for evaluating the functional correctness of code generation models using Python programming problems with unit tests.

Helicone is the all-in-one, open-source LLM developer platform.

HELM (Holistic Evaluation of Language Models) provides tools for the holistic evaluation of language models, including standardized datasets, a unified API for various models, diverse metrics, r, and fairness perturbations, a prompt construction framework, and a proxy server for unified model access.

Inspect is a framework for large language model evaluations.

IsaacLab-Arena is an open-source extension to NVIDIA Isaac Lab for composable environment creation and robotic policy evaluation at scale.

JiWER is a simple and fast python package to evaluate an automatic speech recognition system.

Laminar is an open-source platform to trace, evaluate, label, and analyze LLM data for AI products.

Langfuse is an observability & analytics solution for LLM-based applications.

LangTest is a comprehensive evaluation toolkit for NLP models.

Language Model Evaluation Harness is a framework to test generative language models on a large number of different evaluation tasks.

LangWatch is a visual interface for DSPy and a complete LLM Ops platform for monitoring, experimenting, measuring and improving LLM pipelines, with a fair-code distribution model.

Latitude is an open-source platform for AI agent observability with semantic trace search and issue tracking.

LightEval is a lightweight LLM evaluation suite.

LLMPerf is a tool for evaluating the performance of LLM APIs.

lmms-eval is an evaluation framework meticulously crafted for consistent and efficient evaluation of LMM.

Melting Pot is a suite of test scenarios for multi-agent reinforcement learning.

Meta-World is an open-source simulated benchmark for meta-reinforcement learning and multi-task learning consisting of 50 distinct robotic manipulation tasks.

mireval is a Python library which provides a transparent, standardized, and straightforward way to evaluate Music Information Retrieval systems.

MLPerf Inference is a benchmark suite for measuring how fast systems can run models in a variety of deployment scenarios.

Massive Text Embedding Benchmark (MTEB) is a comprehensive evaluation framework that assesses the performance of text embedding models across diverse tasks and languages, encompassing 8 embedding tasks, 58 datasets, and 112 languages.

NannyML is a library that allows you to estimate post-deployment model performance (without access to targets), detect data drift, and intelligently link data drift alerts back to changes in model performance.

The Open Graph Benchmark (OGB) is a collection of benchmark datasets, data loaders, and evaluators for graph machine learning.

Ollama Grid Search automates the process of selecting the best models, prompts, or inference parameters for a given use-case, allowing you to iterate over their combinations and to visually inspect the results.

onWatch is a lightweight Go CLI that tracks AI API quota usage across multiple providers (Anthropic Pro/Max Plans, Codex, Gemini CLI, Synthetic, Z.ai, GitHub Copilot, MiniMax Coding/Token Plan, Antigravity, OpenRouter) in real time, with consumption rate projections, historical usage graphs, and per-cycle tracking.

OpenCompass is an LLM evaluation platform, supporting a wide range of models (LLaMA, LLaMa2, ChatGLM2, ChatGPT, Claude, etc) over 50+ datasets.

OpenLIT is an open-source AI engineering platform that simplifies LLM workflows with observability, monitoring, guardrails, evaluations, and seamless integrations.

OpenLLMetry provides developers with deep visibility into Large Language Model applications through performance monitoring, execution tracing, and debugging capabilities.

Opik is an open-source platform for evaluating, testing and monitoring LLM applications.

Overcooked-AI is a benchmark environment for fully cooperative human-AI task performance, based on the wildly popular video game Overcooked.

Phoenix is an open-source AI observability platform designed for experimentation, evaluation, and troubleshooting.

PromptBench is a unified evaluation framework for large language models

LLM red teaming and evaluation framework for testing jailbreaks, prompt injection, and other vulnerabilities with CI/CD integration.

RagaAI Catalyst is a comprehensive platform designed to enhance the management and optimization of LLM projects.

Prometheus-Eval is a collection of tools for training, evaluating, and using language models specialized in evaluating other language models.

Ragas is a framework to evaluate RAG pipelines.

RewardBench is a benchmark designed to evaluate the capabilities and safety of reward models.

RLBench is an ambitious large-scale benchmark and learning environment designed to facilitate research in a number of vision-guided manipulation research areas, including: reinforcement learning, imitation learning, multi-task learning, geometric computer vision, and in particular, few-shot learning.

SimplerEnv is a simulated manipulation policy evaluation environments for real robot setups.

SwanLab is an AI training tracking and visualization tool.

Speech-to-Text Benchmark is a minimalist and extensible framework for benchmarking different speech-to-text engines.

TensorFlow Model Analysis (TFMA) is a library for evaluating TensorFlow models on large amounts of data in a distributed manner, using the same metrics defined in their trainer.

TorchBench is a collection of open source benchmarks used to evaluate PyTorch performance.

TruLens provides a set of tools for evaluating and tracking LLM experiments.

TrustLLM is a comprehensive framework to evaluate the trustworthiness of large language models, which includes principles, surveys, and benchmarks.

VBench is a comprehensive benchmark suite for video generative models.

VLMEvalKit is an open-source evaluation toolkit of large vision-language models (LVLMs).

An open-source bias audit toolkit for data scientists, machine learning researchers, and policymakers to audit machine learning models for discrimination and bias, and to make informed and equitable decisions around developing and deploying predictive risk-assessment tools.

Interpretability and explainability of data and machine learning models including a comprehensive set of algorithms that cover different dimensions of explanations along with proxy explainability metrics.

A comprehensive set of fairness metrics for datasets and machine learning models, explanations for these metrics, and algorithms to mitigate bias in datasets and models.

Alibi is an open source Python library aimed at machine learning model inspection and interpretation. The initial focus on the library is on black-box, instance based model explanations.

model interpretability and understanding library for PyTorch developed by Facebook. It contains general purpose implementations of integrated gradients, saliency maps, smoothgrad, vargrad and others for PyTorch models.

Fairlearn is a python toolkit to assess and mitigate unfairness in machine learning models.

InterpretML is an open-source package for training interpretable models and explaining blackbox systems.

A python framework for self-supervised learning on images. The learned representations can be used to analyze the distribution in unlabeled data and rebalance datasets.

LOFO (Leave One Feature Out) Importance calculates the importances of a set of features based on a metric of choice, for a model of choice, by iteratively removing each feature from the set, and evaluating the performance of the model, with a validation scheme of choice, based on the chosen metric.

A Python package for AutoML on tabular data with feature engineering, hyper-parameters tuning, explanations and automatic documentation.

Quantus is an eXplainable AI toolkit for responsible evaluation of neural network explanations

SHapley Additive exPlanations is a unified approach to explain the output of any machine learning model.

Shapash is a Python library that provides several types of visualization that display explicit labels that everyone can understand.

An easy-to-use interface for expanding understanding of a black-box classification or regression ML model.

Feast (Feature Store) is an open source feature store for machine learning. Feast is the fastest path to manage existing infrastructure to productionize analytic data for model training and online inference.

A virtual featurestore. Plug-&-play with your existing infra. Data Scientist approved. Discovery, Governance, Lineage, & Collaboration just a pip install away. Supports pandas, Python, spark, SQL + integrations with major cloud vendors.

Offline/Online Feature Store for ML (Video).

alibi-detect is a Python package focused on outlier, adversarial and concept drift detection.

Darts is a library for user-friendly forecasting and anomaly detection on time series.

A library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

A Python Toolbox for Scalable Outlier Detection (Anomaly Detection).

TFDV (Tensorflow Data Validation) is a library for exploring and validating machine learning data.

CameraTraps (PyTorch Wildlife) is a collaborative deep learning framework for wildlife image analysis, providing detection and classification models trained on large-scale camera trap datasets.

Deep Lake is a data infrastructure optimized for computer vision.

DeepForest is a Python package for training and predicting individual tree crowns and species from airborne RGB imagery using deep learning.

Detectron2 is Facebook AI Research's next generation library that provides state-of-the-art detection and segmentation algorithms.

KerasCV is a library of modular computer vision oriented Keras components.

Kornia is a differentiable computer vision library built on PyTorch that provides a rich set of differentiable image processing and geometric vision algorithms.

LAVIS is a deep learning library for LAnguage-and-VISion intelligence research and applications.

libcom is an image composition toolbox.

Pretrain computer vision models on unlabeled data for industrial applications.

MMCV is a foundational computer vision library from OpenMMLab that provides essential functionalities like image and video processing, data transformation and augmentation, CNN architectures, and optimized CUDA operations.

SuperGradients is an open-source library for training PyTorch-based computer vision models.

Supervision is a Python library designed for efficient computer vision pipeline management, providing tools for annotation, visualization, and monitoring of models.

VideoSys supports many diffusion models with our various acceleration techniques, enabling these models to run faster and consume less memory.

AutoRAG is a RAG AutoML tool for automatically finds an optimal RAG pipeline for your data.

BGE builds one-stop retrieval toolkit for search and RAG.

Cognita is a RAG framework for building modular and production-ready applications.

DocArray is a library for nested, unstructured, multimodal data in transit, including text, image, audio, video, 3D mesh, etc. It allows deep-learning engineers to efficiently process, embed, search, recommend, store, and transfer multimodal data with a Pythonic API.

EmbedAnything is a minimalist, lightweight, and high-performance embedding pipeline built in Rust for generating embeddings from text, images, audio, PDFs, and other media, with support for dense, sparse, ONNX, and late-interaction embeddings.

Faiss is a library for efficient similarity search and clustering of dense vectors.

fastRAG is a research framework for efficient and optimized retrieval augmented generative pipelines, incorporating state-of-the-art LLMs and Information Retrieval.

GraphRAG is a data pipeline and transformation suite that is designed to extract meaningful, structured data from unstructured text using the power of LLMs.

HippoRAG is a novel retrieval augmented generation (RAG) framework inspired by the neurobiology of human long-term memory that enables LLMs to continuously integrate knowledge across external documents.

JamAI Base is an open-source RAG (Retrieval-Augmented Generation) backend platform that integrates an embedded database (SQLite) and an embedded vector database (LanceDB) with managed memory and RAG capabilities. It features built-in LLM, vector embeddings, and reranker orchestration and management, all accessible through a convenient, intuitive, spreadsheet-like UI and a simple REST API.

LangExtract is a Python library that uses LLMs to extract structured information from unstructured text documents based on user-defined instructions. It processes materials such as clinical notes or reports, identifying and organizing key details while ensuring the extracted data corresponds to the source text.

A simple and fast retrieval-augmented generation framework.

llmware provides a unified framework for building LLM-based applications (e.g, RAG, Agents), using small, specialized models that can be deployed privately, integrated with enterprise knowledge sources safely and securely, and cost-effectively tuned and adapted for any business process.

Mem0 enhances AI assistants and agents with an intelligent memory layer, enabling personalized AI interactions.

NGT provides commands and a library for performing high-speed approximate nearest neighbor searches against a large volume of data in high dimensional vector data space.

Non-Metric Space Library (NMSLIB): An efficient similarity search library and a toolkit for evaluation of k-NN methods for generic non-metric spaces.

An open source vector similarity search engine with extended filtering support.

R2R (RAG to Riches) is a comprehensive platform for building, deploying, and scaling RAG applications with hybrid search, multimodal support, and advanced observability.

RAGFlow is a RAG engine based on deep document understanding.

RAGxplorer is a tool to build RAG visualisations.

RAG-FiT is a library designed to improve LLMs ability to use external information by fine-tuning models on specially created RAG-augmented datasets.

TextWorld is a text-based game generator and extensible sandbox learning environment for training and testing reinforcement learning (RL) agents.

Vanna is a RAG framework for SQL generation and related functionality.

Zvec is an open-source, in-process vector database for low-latency similarity search.

aisuite is a simple, unified interface to multiple generative AI providers.

Align-Anything aims to align any modality large models (any-to-any models), including LLMs, VLMs, and others, with human intentions and values

BERTopic is a topic modeling technique that leverages transformers and c-TF-IDF to create dense clusters allowing for easily interpretable topics whilst keeping important words in the topic descriptions.

Burr helps you develop applications that make decisions (chatbot, agent, simulation). It comes with production-ready features (telemetry, persistence, deployment, etc.) and the open-source, free, and local-first Burr UI.

CodeTF is a one-stop Python transformer-based library for code large language models (Code LLMs) and code intelligence, provides a seamless interface for training and inferencing on code intelligence tasks like code summarization, translation, code generation and so on.

Context7 provides up-to-date code docs for prompts and AI coding agents.

Dify is an open-source LLM app development platform whose intuitive interface combines agentic AI workflow, RAG pipeline, agent capabilities, model management, observability features and more, letting you quickly go from prototype to production.

A framework for programming with foundation models.

Dust assists in the design and deployment of large language model apps.

ESPnet is an end-to-end speech processing toolkit.

FastChat is an open platform for training, serving, and evaluating large language model based chatbots.

Simple framework for state-of-the-art NLP developed by Zalando which builds directly on PyTorch.

FunASR is a production-grade ASR toolkit supporting 50+ languages with built-in VAD, punctuation, speaker diarization, and emotion recognition, plus Docker/WebSocket/REST deployment and ONNX runtime support.

LLM-based ASR supporting 31 languages including Chinese dialects with native punctuation, timestamps, and speaker diarization.

Gensim is a Python library for topic modelling, document indexing and similarity retrieval with large corpora.

Simple and efficient pytorch-native transformer text generation.

h2oGPT is an open source generative AI, gives organizations like yours the power to own large language models while preserving your data ownership.

Haystack is an open source NLP framework to interact with your data using Transformer models and LLMs (GPT-3 and alike). Haystack offers production-ready tools to quickly build ChatGPT-like question answering, semantic search, text generation, and more.

ICE is a Python library and trace visualizer for language model programs.

Jan is an open-source ChatGPT alternative that runs 100% offline on your computer, allowing you to download and run LLMs locally with full control and privacy.

Lamini is an LLM engine for rapidly customizing models.

LangChain assists in building applications with LLMs through composability.

LlamaIndex (GPT Index) is a data framework for your LLM application.

LLaMA is intended as a minimal, hackable and readable example to load LLaMA (arXiv) models and run inference.

LLaMA-Factory makes it easy to fine-tunes 100+ large language models with zero-code CLI and Web UI

LLMBox is a comprehensive library for implementing LLMs, including a unified training pipeline and comprehensive model evaluation.

LLaMA2-Accessory is an open-source toolkit for pretraining, finetuning and deployment of Large Language Models (LLMs) and multimodal LLMs.

LMFlow is an extensible, convenient, and efficient toolbox for finetuning large machine learning models.

Megatron-LM is a highly optimized and efficient library for training large language models.

MindNLP is an easy-to-use and high-performance NLP and LLM framework based on MindSpore, compatible with models and datasets of Huggingface.

MLC LLM is a universal solution that allows any language models to be deployed natively on a diverse set of hardware backends and native applications, plus a productive framework for everyone to further optimize model performance for their own use cases.

MLX LM is a Python package for generating text and fine-tuning large language models on Apple silicon with MLX, featuring integration with Hugging Face Hub and support for quantization and distributed inference.

Get up and running with large language models, locally.

olmOCR is a toolkit for training language models to work with PDF documents in the wild.

PaddleNLP is a Large Language Model (LLM) development suite based on the PaddlePaddle deep learning framework, supporting efficient large model training, lossless compression, and high-performance inference on various hardware devices.

Promptise Foundry is a production Python framework for agentic AI and MCP servers, covering autonomous runtime, memory, tool integration, governance (budget, health, mission, secrets), guardrails, semantic caching, and observability.

PyLLMs is a minimal Python library to connect to various Language Models (LLMs) with a built-in model performance benchmark.

Semantic Kernel is an SDK that integrates Large Language Models (LLMs) like OpenAI, Azure OpenAI, and Hugging Face with conventional programming languages like C#, Python, and Java. Semantic Kernel achieves this by allowing you to define plugins that can be chained together in just a few lines of code.

Sentence Transformers provides an easy method to compute dense vector representations for sentences, paragraphs, and images.

spaCy is a library for advanced Natural Language Processing in Python and Cython.

SWIFT is a scalable lightweight infrastructure for deep learning model fine-tuning.

A framework for building neural networks in Tensorflow, particularly sequence models.

TensorFlow Text provides a collection of text related classes and ops ready to use with TensorFlow 2.0.

ToolBench is an open platform for training, serving, and evaluating large language model for tool learning.

Huggingface's library of state-of-the-art pretrained models for Natural Language Processing (NLP).

EasyRec is a framework for large scale recommendation algorithms.

Gorse aims to be a universal open-source recommender system that can be quickly introduced into a wide variety of online services.

NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.

Recommenders contains benchmark and best practices for building recommendation systems, provided as Jupyter notebooks.

TorchRec is a PyTorch domain library built to provide common sparsity and parallelism primitives needed for large-scale recommender systems (RecSys).

Acme is a library of reinforcement learning (RL) building blocks that strives to expose simple, efficient, and readable agents.

AReaL is a reinforcement learning library.

ChatLearn is a flexible and efficient reinforcement learning training framework for large language models, supporting distributed training engines (FSDP2, Megatron) and inference engines (vLLM, SGLang) with modern RL algorithms such as GRPO and GSPO.

CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch.

CompilerGym is a library of easy to use and performant reinforcement learning environments for compiler tasks.

d3rlpy is an offline deep reinforcement learning library for practitioners and researchers.

D4RL is an open-source benchmark for offline reinforcement learning.

Dopamine is a research framework for fast prototyping of reinforcement learning algorithms. It aims to fill the need for a small, easily grokked codebase in which users can freely experiment with wild ideas (speculative research).

EvoTorch is an open source evolutionary computation library developed at NNAISENSE, built on top of PyTorch.

FinRL is the first open-source framework to demonstrate the great potential of financial reinforcement learning.

Gymnasium is an open source Python library for developing and comparing reinforcement learning algorithms by providing a standard API to communicate between learning algorithms and environments, as well as a standard set of environments compliant with that API.

Gymnasium-Robotics contains a collection of Reinforcement Learning robotic environments that use the Gymansium API. The environments run with the MuJoCo physics engine and the maintained mujoco python bindings.

Jumanji is a suite of Reinforcement Learning (RL) environments written in JAX providing clean, hardware-accelerated environments for industry-driven research.

MARLlib is a comprehensive Multi-Agent Reinforcement Learning algorithm library based on RLlib. It provides MARL research community with a unified platform for building, training, and evaluating MARL algorithms.

Mava is a framework for distributed multi-agent reinforcement learning in JAX.

Melting Pot is a suite of test scenarios for multi-agent reinforcement learning.

MetaDrive is a driving simulator that composes diverse driving scenarios for generalizable RL.

The Minigrid library contains a collection of discrete grid-world environments to conduct research on Reinforcement Learning. The environments follow the Gymnasium standard API and they are designed to be lightweight, fast, and easily customizable.

MiniWorld is a minimalistic 3D interior environment simulator for reinforcement learning & robotics research.

ML-Agents is an open-source project that enables games and simulations to serve as environments for training reinforcement learning intelligent agents.

MLGym is a gym environment enabling research on reinforcement learning (RL) algorithms for training such agents for ML tasks.

MushroomRL is a Python reinforcement learning (RL) library whose modularity allows to easily use well-known Python libraries for tensor computation (e.g. PyTorch, Tensorflow) and RL benchmarks (e.g. OpenAI Gym, PyBullet, Deepmind Control Suite).

OmniSafe is an infrastructural framework designed to accelerate safe reinforcement learning (RL) research.

OpenRLHF is an open-source framework for reinforcement learning from human feedback (RLHF).

PARL is a flexible and high-efficient reinforcement learning framework.

PettingZoo is a Python library for conducting research in multi-agent reinforcement learning, akin to a multi-agent version of Gymnasium.

ranx is a library of fast ranking evaluation metrics implemented in Python, leveraging Numba for high-speed vector operations and automatic parallelization.

RL4CO is a PyTorch library for all things reinforcement learning for combinatorial optimization (CO).

RL2 is a reinforcement learning library.

RLinf is a reinforcement learning library.

ROLL is a reinforcement learning library.

skrl is an open-source modular library for Reinforcement Learning written in Python (using PyTorch) and designed with a focus on readability, simplicity, and transparency of algorithm implementation.

SkyRL is a full-stack reinforcement learning library providing modular training frameworks, cross-platform inference backends, agentic pipelines, and gymnasium environments for long-horizon, real-world RL tasks.

slime is an LLM post-training framework for RL Scaling.

A fork of OpenAI Baselines, implementations of reinforcement learning algorithms.

A reliable, scalable and easy to use TensorFlow library for contextual bandits and reinforcement learning.

TorchRL is an open-source Reinforcement Learning (RL) library for PyTorch.

Train transformer language models with reinforcement learning.

veRL (HybridFlow) is a flexible, efficient and industrial-level RL(HF) training framework designed for LLMs.

AI2-THOR is a near photo-realistic interactable framework for AI agents.

Genesis is a physics platform for Embodied AI and robot simulation.

Habitat-Sim is a flexible, high-performance 3D simulator for Embodied AI research.

IsaacLab is a unified and modular framework for robot learning that leverages NVIDIA Isaac Sim.

LeRobot provides models, datasets, and tools for real-world robotics and imitation learning.

robosuite is a simulation framework powered by the MuJoCo physics engine for robot learning.

RoboVerse is a comprehensive robotics simulation platform with diverse environments.

Apache ECharts is a powerful, interactive charting and data visualization library for browser.

A modern, enterprise-ready business intelligence web application.

Bokeh is an interactive visualization library for Python that enables beautiful and meaningful visual presentation of data in modern web browsers.

A VS Code extension for viewing and exploring large machine learning datasets (CSV, JSON, Parquet, etc.) directly within the editor without the IDE crashing.

A VS Code extension to view Weights & Biases experiments, logs, and artifacts within the IDE, eliminating the need to switch to the web UI & preserving data privacy by being 100% offline.

Transform data and create rich visualizations iteratively with AI.

An implementation of the grammar of graphics for R.

Quickly create and share demos of models - by only writing Python. Debug models interactively in your browser, get feedback from collaborators, and generate public links without deploying anything.

Kangas is a tool for exploring, analyzing, and visualizing large-scale multimedia data. It provides a straightforward Python API for logging large tables of data, along with an intuitive visual interface for performing complex queries against your dataset.

A Python 2D plotting library which produces publication-quality figures in a variety of hardcopy formats and interactive environments across platforms.

Model Explorer is a tool for visualizing and exploring machine learning models, providing an intuitive graph-based view to understand model structure, inspect layer details, and navigate large neural networks.

Netron is a viewer for neural network, deep learning and machine learning models.

Streaming pivot visualization via WebAssembly.

An interactive, open source, and browser-based graphing library for Python.

Redash is anopen source visualisation framework that is built to allow easy access to big datasets leveraging multiple backends.

Rerun is an open-source SDK for logging, storing, querying, and visualizing multimodal data, designed for robotics, computer vision, and spatial AI.

Seaborn is a Python visualization library based on matplotlib. It provides a high-level interface for drawing attractive statistical graphics.

Spotlight helps you to identify critical data segments and model failure modes. It enables you to build and maintain reliable machine learning models by curating high-quality datasets.

Streamlit lets you create apps for your machine learning projects with deceptively simple Python scripts. It supports hot-reloading, so your app updates live as you edit and save your file.

Write TensorBoard events with simple function call.

TensorBoard is a visualization toolkit for machine learning experimentation that makes it easy to host, track, and share ML experiments.

Torchvista is an interactive notebook-based tool to visualise the forward pass of any PyTorch model as a computation graph within notebooks, with support for collapsible nested modules and error-tolerant partial visualisations.

Transformer Explainer is an interactive visualization tool designed to help anyone learn how Transformer-based models like GPT work.

Vega-Altair is a declarative statistical visualization library for Python.

ydata-profiling provides a one-line Exploratory Data Analysis (EDA) experience in a consistent and fast solution.

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.

Apache Atlas framework is an extensible set of core foundational governance services – enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop and allows integration with the whole enterprise data ecosystem.

DataHub is LinkedIn's generalized metadata search & discovery tool.

Marquez is an open source metadata service for the collection, aggregation, and visualization of a data ecosystem's metadata.

Metacat is a unified metadata exploration API service. Metacat focuses on solving these problems: 1) federated views of metadata systems; 2) arbitrary metadata storage about data sets; 3) metadata discovery.

a library for recording and retrieving metadata associated with ML developer and data scientist workflows.

A super-easy way to record, search and compare AI experiments.

Auto-Magical Experiment Manager & Version Control for AI (previously Trains).

DataHub is an open-source data catalog for the modern data stack.

Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository.

DVC (Data Version Control) is a git fork that allows for version management of models.

HuggingFace Model Downloader is a utility tool for downloading models and datasets from the HuggingFace website. It offers multithreaded downloading for LFS files and ensures the integrity of downloaded models with SHA256 checksum verification.

Version control for machine learning.

KitOps is an open and standards-based packaging and versioning system for AI/ML projects that works with all the AI/ML, development, and DevOps tools you are already using.

Repeatable, atomic and versioned data lake on top of object storage.

Open source platform to manage the ML lifecycle, including experimentation, reproducibility and deployment.

Neptune is a scalable experiment tracker for teams that train foundation models.

A platform for reproducible and scalable machine learning and deep learning on kubernetes - (Video).

Versioning, reproducibility and deployment of data and models.

Tool to help you configure, organize, log and reproduce machine learning experiments.

A graph database management system that stores data like git.

Weights & Biase is a machine learning experiment tracking, dataset versioning, hyperparameter search, visualization, and collaboration.

AutoTrain Advanced is a no-code solution that allows you to train machine learning models in just a few clicks.

Avalanche is an end-to-end Continual Learning library to provide a shared and collaborative open-source (MIT licensed) codebase for fast prototyping, training and reproducible evaluation of continual learning algorithms.

Axolotl is a tool designed to streamline the fine-tuning of various AI models, offering support for multiple configurations and architectures.

BindsNET is a spiking neural network simulation library geared towards the development of biologically inspired algorithms for machine learning.

Continuous Machine Learning (CML) is an open-source library for implementing continuous integration & delivery (CI/CD) in machine learning projects.

CoreNet is a deep neural network toolkit that allows researchers and engineers to train standard and novel small and large-scale models for variety of tasks, including foundation models (e.g., CLIP and LLM), object classification, object detection, and semantic segmentation.

DataLinter is an open-source contextual linter for data and code, designed to be data and code agnosting through plugins.

Deep learning training platform with integrated support for distributed training, hyperparameter tuning, and model management (supports Tensorflow and Pytorch).

dstack is an open-source container orchestrator that simplifies workload orchestration and drives GPU utilization for ML teams.

Machine learning development environment for data science and AI/ML engineering teams.

Fairseq(-py) is a sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling and other text generation tasks.

The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications.

Fast scalable Machine Learning platform for smarter applications: Deep Learning, Gradient Boosting & XGBoost, Random Forest, Generalized Linear Modeling (Logistic Regression, Elastic Net), K-Means, PCA, Stacked Ensembles, Automatic Machine Learning (AutoML), etc..

Hopsworks is a data-intensive platform for the design and operation of machine learning pipelines.

Ignite is a high-level library to help with training and evaluating neural networks in PyTorch flexibly and transparently.

A cloud-native platform for machine learning based on Google's internal machine learning pipelines.

Ludwig is a low-code framework for building custom AI models like LLMs and other deep neural networks.

MFTCoder is an open-source project of CodeFuse for accurate and efficient Multi-task Fine-tuning(MFT) on Large Language Models(LLMs), especially on Code-LLMs(large language model for code tasks).

Standardisation of pipeline and model serialization for Spark, Tensorflow and sklearn.

Nanotron provides distributed primitives to train a variety of models efficiently using 3D parallelism.

NVIDIA NeMo is a scalable and cloud-native generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains. It is designed to help you efficiently create, customize, and deploy new generative AI models by leveraging existing code and pre-trained model checkpoints.

Prime is a framework for efficient, globally distributed training of AI models over the internet.

) - low-code library for training and deploying models (scikit-learn, XGBoost, LightGBM, spaCy)

Platform to build resource-intensive pipelines with simple Python.

Skaffold is a command line tool that facilitates continuous development for Kubernetes applications. You can iterate on your application source code locally then deploy to local or remote Kubernetes clusters.

Tensorflow Extended (TFX) is a production oriented configuration framework for ML based on TensorFlow, incl. monitoring and model version management.

Fine-tuning & Reinforcement Learning for LLMs. Train OpenAI gpt-oss, DeepSeek-R1, Qwen3, Gemma 3, TTS 2x faster with 70% less VRAM.

AutoAWQ is an easy-to-use package for 4-bit quantized models.

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Activation-aware Weight Quantization for LLM Compression and Acceleration.

GGML is a high-performance, tensor library for machine learning that enables efficient inference on CPUs, particularly optimized for large language models.

Intel® Neural Compressor aims to provide popular model compression techniques such as quantization, pruning (sparsity), distillation, and neural architecture search on mainstream frameworks.

Neural Network Exchange Format (NNEF) is an open standard for representing neural network models to enable interoperability and portability across different machine learning frameworks and platforms.

ONNX (Open Neural Network Exchange) is an open-source format designed to facilitate interoperability and portability of machine learning models across different frameworks and platforms.

PFA (Portable Format for Analytics) format is a standard for representing and exchanging predictive models and analytics workflows in a portable, JSON-based format.

PMML (Predictive Model Markup Language) is an XML-based standard for representing and sharing predictive models between different applications.

Quanto aims to simplify quantizing deep learning models.

The AI Gateway is a blazing fast AI Gateway with integrated guardrails.

ART (Adversarial Robustness Toolbox) provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference.

CipherChat is a framework to evaluate the generalization capability of safety alignment for LLMs

DeepTeam is a simple-to-use, open-source LLM red teaming framework, for penetration testing and safe guarding large-language model systems.

FATE (Federated AI Technology Enabler) is the world's first industrial grade federated learning open source framework to enable enterprises and institutions to collaborate on data while protecting data security and privacy.

FedML provides a research and production integrated edge-cloud platform for Federated/Distributed Machine Learning at anywhere at any scale.

Flower is a Federated Learning Framework with a unified approach. It enables the federation of any ML workload, with any ML framework, and any programming language.

This is a C++ library of ε-differentially private algorithms, which can be used to produce aggregate statistics over numeric data sets containing private or sensitive information.

Guardrails is a package that lets a user add structure, type and quality guarantees to the outputs of large language models.

NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems.

Opacus is a library that enables training PyTorch models with differential privacy. It supports training with minimal code changes required on the client, has little impact on training performance, and allows the client to online track the privacy budget expended at any given moment.

OpenFL is a Python framework for Federated Learning. OpenFL is designed to be a flexible, extensible and easily learnable tool for data scientists. OpenFL is developed by Intel Internet of Things Group (IOTG) and Intel Labs.

A Python library for secure, private Deep Learning. PySyft decouples private data from model training, using Multi-Party (MPC) within PyTorch.

A Python library that includes implementations of TensorFlow optimizers for training machine learning models with differential privacy.

A Framework for Confidential Machine Learning on Encrypted Data in TensorFlow.