An open-source inference gateway for production LLM deployments, developed by the LightSeek Foundation. It puts self-hosted inference engines such as vLLM, SGLang, and TensorRT-LLM alongside cloud APIs such as OpenAI, Anthropic, and Gemini behind a single OpenAI-compatible endpoint. The core idea is that your application keeps calling “the OpenAI API,” while the gateway decides which model and which server each request actually goes to. SMG is written in Rust and offers KV-cache-aware routing, automatic failover through circuit breakers, and observability via Prometheus metrics. It is licensed under Apache 2.0 and can be installed through pip, Cargo, Docker, or Helm.
Key Features
- A single OpenAI-compatible endpoint: Self-hosted engines (HTTP / gRPC) and cloud providers share one entry point. You can swap backends, or run several side by side, without rewriting application code
- Cache-aware routing: SMG tracks the KV-cache state held by workers such as vLLM, SGLang, and TensorRT-LLM, and steers requests that share context to the same worker to improve GPU utilization. Besides cache-aware routing, policies include least-load, power-of-two, consistent hashing, and round robin
- Traffic control and automatic failover: Rate limiting, load balancing, and circuit breakers are built in, so an unhealthy backend can be pulled out of rotation and traffic redirected elsewhere
- Observability: More than 90 Prometheus metrics, OpenTelemetry tracing, and structured JSON logs. You can see how much traffic each model receives and where latency accumulates from your existing monitoring stack
- Multi-tenancy and enterprise features: API-key and OIDC authentication, tenant isolation, priority-based admission scheduling, and WebAssembly plugins for extension
- MCP tool execution: Model Context Protocol tool discovery and execution over stdio, SSE, and HTTP, with configurable approval policies
- Kubernetes-native: Pod discovery via label selectors, so the gateway follows workers as they scale up and down inside a cluster
Pricing
| Plan | Price | Highlights |
|---|---|---|
| Open source | $0 (Apache 2.0) | All features available when self-hosted. Install via pip / Cargo / Docker / Helm |
Neither the official site nor the documentation mentions a commercial edition, an enterprise edition, or paid support. In practice, your cost is driven by the server running the gateway plus the GPU instances and metered cloud APIs behind it.
Pricing is current as of August 2026. Please check the official site for the latest information.
Pros & Cons
✅ Pros
- Because it is OpenAI-compatible, you can start by simply pointing an existing application at a different endpoint
- Self-hosted models and cloud APIs share one entry point, which makes it a natural place to optimize cost or move workloads in-house step by step
- Cache-aware routing tends to use GPUs more efficiently than plain round robin
- Metrics go straight into Prometheus and OpenTelemetry, so it slots into an existing monitoring stack
- Apache 2.0 open source, which avoids vendor lock-in
⚠️ Cons
- It is an infrastructure component and generates nothing on its own; you still need inference engines or cloud APIs behind it
- Deployment and operations assume familiarity with Kubernetes, load balancing, and metrics monitoring. It is heavy for one person trying a model on a single machine
- Unlike a SaaS gateway, you are responsible for the redundancy, upgrades, and incident response of the gateway itself
- Released in February 2026, so long-term operational case studies and community know-how are still accumulating
Comparison with Similar Services
| Criteria | SMG | LiteLLM | Portkey | Envoy AI Gateway |
|---|---|---|---|---|
| Delivery | Open source (self-hosted) | Open source + cloud | SaaS-first (OSS edition available) | Open source (self-hosted) |
| Implementation | Rust | Python | ─ | Go / Envoy |
| Main strength | Cache-aware routing for self-hosted engines | Breadth of provider support, easy setup | Guardrails and analytics for operations | Integration with Envoy infrastructure |
| Self-hosted engines | vLLM / SGLang / TensorRT-LLM / MLX and more | Through OpenAI-compatible servers | Mainly aimed at cloud APIs | Through OpenAI-compatible servers |
| Target scale | Production deployments with GPU clusters | Small to mid-size, including prototypes | Teams and companies | Organizations already running Envoy |
Who Is It For
- Teams running vLLM or SGLang on their own GPUs who want a serious routing layer in front of them
- Product developers who mix cloud APIs with self-hosted models and want to route by cost or load
- Infrastructure engineers who want LLM usage exposed as metrics and handled by the same operational workflow as the rest of their systems
- Organizations offering an internal LLM platform to multiple tenants, with per-tenant authentication and priority control
- Companies that want to avoid vendor lock-in and build their platform on Apache 2.0 open source
Summary
SMG behaves less like an AI product and more like a piece of network equipment for LLM inference traffic. It generates nothing by itself, but it gives you one place to decide how requests flow through a production environment where self-hosted engines and cloud APIs coexist. Cache-aware routing and the large set of Prometheus metrics start to pay off once GPU cost and latency both matter. A reasonable starting point is to bring up a single node with pip or Docker and repoint an existing application at it. Conversely, if you are one person experimenting with a single model, you do not need this layer yet.