What are we doing? Presenting one consistent interface to application code, regardless of which provider actually handles a request. Why? Without this, provider-specific logic ends up duplicated across every calling application.

class Provider:
    def call(self, request): ...

class ProviderA(Provider):
    def call(self, request):
        return provider_a_sdk.generate(request.to_provider_a_format())

class ProviderB(Provider):
    def call(self, request):
        return provider_b_sdk.complete(request.to_provider_b_format())

What are we doing? Choosing which model handles a request, based on task type rather than a hardcoded choice in application code. Why? Simple tasks can use a cheaper, faster model; complex reasoning may need a stronger one — this is a cost and latency lever, not just a technical choice.

def select_provider(task_type):
    if task_type == "classification":
        return ProviderA()  # fast, cheap
    if task_type == "reasoning":
        return ProviderB()  # stronger, slower
    return ProviderA()

What are we doing? Retrying a failed call, and falling back to an alternate provider if the primary one is unavailable. Why? Without a fallback, a single provider outage becomes a full outage for every application behind the gateway.

def handle_request(request):
    provider = select_provider(request.task_type)
    try:
        return call_with_retry(provider, request, max_retries=2, timeout=request.timeout)
    except ProviderUnavailable:
        fallback = get_fallback(provider)
        return call_with_retry(fallback, request, max_retries=1, timeout=request.timeout)
    except AllProvidersFailed:
        return degraded_response(request)

What are we doing? Bounding how much any one caller can use, and recording enough to understand spend. Why? Without limits, one misbehaving caller can exhaust shared provider quota; without tracking, cost problems are only discovered when the bill arrives.

What are we doing? Logging enough per request to debug and track cost, without capturing unnecessary sensitive content. Why? The gateway sees every request — it's the natural place to centralize this visibility, but also a common place for sensitive data to leak if logged indiscriminately.

• No fallback when the primary provider fails

  A single provider outage becomes a full outage for every application behind the gateway.

• No per-caller rate limiting

  One misbehaving caller can exhaust shared provider quota for everyone.

• Logging full prompts and responses without redaction

  Gateway logs are a common place for sensitive content to leak.

• Hardcoding model choice in application code

  This defeats the purpose of routing — the decision should live in the gateway.

• Making the gateway itself a single point of failure

  It needs its own reliability story, since every request now depends on it.