An LLM gateway is a shared layer that routes requests to model providers while handling auth, logging, and fallback.

Streaming returns a model's output incrementally as it's generated instead of waiting for the full response.

Caching stores previous results so repeated or similar requests can be served faster and more cheaply.

Rate limiting restricts how many requests a client can make in a given time window to protect a system from overload.

Observability is the ability to see what an AI system is doing in production through logs, traces, and metrics.

Evaluation systematically measures the quality of a model's or system's outputs against defined criteria.

Guardrails are checks that constrain model input or output to prevent unsafe, incorrect, or off-policy behavior.

Cost optimization reduces the expense of running AI systems through techniques like caching, routing, and smaller models where appropriate.

AI security protects AI systems from risks such as prompt injection, data leakage, and misuse of tools.