Why OpenAI Compatibility Matters
You've trained a model. It works. Now you need to serve it. The fastest path to production isn't building a bespoke API — it's exposing an endpoint that speaks the same language as OpenAI's Chat Completions API. Every SDK, every framework, every piece of example code on the internet already knows this contract. Match it, and your model becomes a drop-in replacement.
What the Contract Actually Requires
The OpenAI Chat Completions endpoint expects a POST to /v1/chat/completions with a JSON body containing model, messages (an array of role/content objects), and optional parameters like temperature, max_tokens, stream, and tools. The response mirrors this structure with choices, usage, and metadata fields. Streaming adds Server-Sent Events with data: prefixes and a final [DONE] marker. That's the entire surface area.
Choosing a Serving Engine
- vLLM — The current default for throughput. PagedAttention handles long contexts efficiently, and the built-in OpenAI-compatible server starts with one flag:
vllm serve.--api-key - TGI (Text Generation Inference) — Hugging Face's production server. Excellent for continuous batching and tensor parallelism across GPUs. OpenAI compatibility ships in
--api-type openai. - llama.cpp server — CPU-first, runs quantized GGUF models with surprising speed. The
--openai-compatflag enables the endpoint. - FastAPI + custom wrapper — Full control, but you own the streaming logic, tokenization, and batching. Only choose this when the above don't fit.
Authentication and Multi-Model Routing
Production endpoints need auth. The simplest approach: a static bearer token validated by middleware. For multi-model setups, map the model field in the request to different loaded weights or LoRA adapters. vLLM supports this with --model aliases; TGI uses a router in front. Keep the mapping table in a config file, not hardcoded.
Streaming Done Right
Streaming breaks when the proxy or load balancer buffers responses. Disable buffering: proxy_buffering off in Nginx, proxy_read_timeout high enough for slow tokens. Send proper SSE headers: Content-Type: text/event-stream, Cache-Control: no-cache, Connection: keep-alive. Each chunk is data: {...}
. The final chunk is data: [DONE]
.
Observability You'll Actually Use
- Request latency (p50, p95, p99) per model
- Tokens per second — the true throughput metric
- Queue depth — when requests wait for GPU slots
- Error rate by type (OOM, validation, timeout)
- Cost per 1K tokens if you're metering
Prometheus + Grafana works. So does a structured JSON log line per request shipped to Loki or Elasticsearch.
Common Pitfalls
- Ignoring
stopsequences — Clients send them. Honor them or generation runs long. - Breaking on
tools/function_call— Even if your model doesn't support tools, return a valid empty response instead of 500. - Hardcoding
modelin responses — Echo the requested model name; clients assert on it. - No request ID — Add
x-request-idto responses. Debugging distributed traces without it is miserable.
Going to Production
Put the server behind a reverse proxy (Nginx, Caddy, Traefik) with TLS termination. Enable rate limiting per API key. Health-check /health or /v1/models for orchestration. Deploy multiple replicas behind a load balancer with sticky sessions for streaming. Set resource limits: GPU memory fraction, max concurrent requests, request timeout. Then load test with realistic payloads — not "hello".
