A Model Gateway That Trains Its Router on Your Own Traces

My notes

Summary

Experiential is an open source model gateway and router for agent workflows: one OpenAI-compatible API across hosted, BYOK, and local or custom LLMs, with per-user and per-agent access controls and spend budgets. It also turns production traffic, captured as OpenTelemetry traces, into a custom router or fine-tuned model optimized for quality, speed, and cost, available both as a local CLI and a hosted platform.

Key Insight

• It goes beyond a simple proxy: exp build <project> ingests OTLP traces and simulates and optimizes a router against that real traffic, so routing decisions are trained on the agent’s actual usage patterns rather than generic benchmarks.
• Closes the loop from observability to a bespoke model. exp optimize model <project> fine-tunes an open source model you own using Tinker, driven by the same collected traces.
• Same API surface locally and hosted: pip install experiential && exp runs a local OpenAI-compatible gateway, and the hosted endpoint is a drop-in equivalent (OpenAI-compatible plus Anthropic Messages API), so teams can start local and move to managed without changing client code.
• Cost governance is a first-class onboarding step, not bolted on. The setup wizard shows a default $50 command budget before issuing the one-time key.
• The repo ships copy-paste setup prompts (SETUP.md) explicitly meant to be handed to a coding agent so the agent configures its own gateway access, designed for agent-to-agent onboarding rather than just a human running the CLI.
• Telemetry is anonymous and aggregate via PostHog by default and explicitly excludes prompts, traces, actions, model names, and credentials, which is relevant for privacy review before adoption. Toggle it with exp config telemetry disable/enable.
• The router is embeddable, not just a standalone proxy: exp.load_router("my-project") in Python returns a project-scoped OpenAI client backed by its own private gateway instance.
• Test and dev conventions are standard modern Python tooling (uv sync --extra dev, ruff format/check, ty check, pytest), consistent with a codebase built to be extended.