Every project
Semantic Caching Proxy
A drop-in OpenAI-compatible cache that recognises the same question asked differently.
What it does
Exact-match caching almost never hits on LLM traffic — people phrase the same thing differently every time. This proxy matches on embedding similarity. The hard part is streaming: the writer only caches responses that completed successfully, so a half-finished stream never becomes permanent. Temperature-keyed TTL tiers and Prometheus metrics included.