Skip to main content

What is BentoML?

BentoML is an open-source platform for machine learning model serving that makes it easy to build production-ready ML services. It provides a standardized way to package, deploy, and scale machine learning models in production.

Overview​

BentoML streamlines the entire ML deployment workflow, from model packaging to production deployment. It's designed to be framework-agnostic, supporting all major ML frameworks including TensorFlow, PyTorch, Scikit-learn, XGBoost, and many more.

Key Features​

🚀 Production-Ready

  • Automatically generates REST APIs for your models
  • Built-in support for batching and adaptive inference
  • High-performance model serving with async support
  • Automatic OpenAPI documentation generation

🔧 Framework Agnostic

  • Works with any ML framework (TensorFlow, PyTorch, Scikit-learn, etc.)
  • Custom model implementations supported
  • Pre-built integrations for popular frameworks
  • Easy to extend with custom runners

📦 Easy Packaging

  • Package models with dependencies and code
  • Version control for models
  • Reproducible builds
  • Docker containerization built-in

☁️ Flexible Deployment

  • Deploy locally, on-premises, or to cloud
  • Native Kubernetes support
  • Integration with cloud platforms (AWS, GCP, Azure)
  • BentoCloud for managed deployments

⚡ Performance Optimized

  • Adaptive batching for throughput optimization
  • Multi-model serving
  • GPU support
  • Request routing and load balancing

How BentoML Works​

BentoML follows a simple workflow:

  1. Save Model - Save your trained model using BentoML's model store
  2. Create Service - Define a service class with inference logic
  3. Build Bento - Package everything into a Bento (model + code + dependencies)
  4. Deploy - Deploy the Bento to your target environment
# Simple example
import bentoml

# Step 1: Save model
bentoml.sklearn.save_model("iris_classifier", model)

# Step 2: Create service
@bentoml.service
class IrisClassifier:
    def __init__(self):
        self.model = bentoml.sklearn.get("iris_classifier:latest")
    
    @bentoml.api
    def predict(self, input_data):
        return self.model.predict(input_data)

Why Choose BentoML?​

Simplifies ML Operations​

BentoML abstracts away the complexities of production ML serving:

  • No need to manually create REST APIs
  • Automatic input/output validation
  • Built-in monitoring and logging
  • Handles model versioning

Production-Grade Performance​

Optimized for real-world production scenarios:

  • Adaptive batching improves throughput by 10-100x
  • Efficient resource utilization
  • Support for both online and batch inference
  • GPU acceleration support

Cloud-Native Design​

Built for modern cloud infrastructure:

  • Kubernetes-ready containers
  • Horizontal scaling support
  • Integration with service meshes
  • Cloud platform integrations

Use Cases​

BentoML is ideal for:

  • API Services - Serve models through REST/gRPC APIs
  • Batch Inference - Process large datasets efficiently
  • Multi-Model Serving - Serve multiple models in one service
  • Real-Time Predictions - Low-latency inference endpoints
  • Model A/B Testing - Test different model versions
  • Edge Deployment - Deploy to edge devices and IoT

Architecture​

BentoML consists of several key components:

Model Store​

Centralized repository for trained models with version control.

Service Definition​

Python-based service definition using decorators and type hints.

Bento​

A deployable artifact containing model, code, and dependencies.

Runner​

High-performance model inference engine with batching support.

API Server​

Production-ready HTTP/gRPC server for serving predictions.

Getting Started​

Ready to get started with BentoML? Continue to: