Guide
LLM gateway and proxy operations
Three posts from the series Running LLM APIs. The first asks what an LLM gateway takes on when service code stops calling providers itself. The second runs a LiteLLM proxy and records what the Admin UI shows, and what shows up only with Grafana and Langfuse. The third separates prompt caching from a response cache and asks when the cache pays for itself.
The number is the reading order. The date is only when the post was published.
1
LLM API Gateway: OpenRouter, LiteLLM, and Cloudflare
Calling the OpenAI, Anthropic, and Gemini APIs from service code repeats format differences, failover, and key and cost management in every service. The post follows how a gateway pulls that into one layer, what you give up for it, and how OpenRouter, LiteLLM, and Cloudflare AI Gateway compare on latency, fallbacks, response caching, and per-team budgets. The comparison uses official docs and published measurements. Cloudflare added 26–37 ms to the first token. OpenRouter added a little over 200 ms.
2
Monitoring a LiteLLM Proxy - What the Admin UI Covers, and What Only Grafana and Langfuse Show
A LiteLLM proxy with Postgres attached shows who spent how much. Failure rate, latency, and whether fallback actually happens need Prometheus and Grafana before they show up as time series and alerts. Request bodies, sessions, traces, and scores need a tracing tool such as Langfuse. The post runs the same mock traffic through all three for 14 minutes and writes down what each screen shows.
3
When AI Caching Pays Off
A cache hit on an AI API does not hand back a stored answer. It reuses the KV computed for the start of the input. The post works out which requests hit, how high the hit rate has to be to break even once write costs are counted, and compares an agent with a chatbot using Claude Code measurements and production logs from a Slack chatbot. In the agent sessions, 97.6% of input tokens were cache reads.