Global Advisors implemented a shared inference gateway between AI-enabled applications and a changing portfolio of local and external models. The gateway gave applications one stable contract while centralising routing, access, policy, observability and failure behaviour.

The technical work mattered because direct model integration creates hidden coupling. Model names, credentials, provider-specific payloads, context limits, availability and economics leak into every application. A gateway changes model choice from an application rewrite into a governed operating decision.

Maturity: active internal control plane, with policy, evaluation and operational practices continuing to mature.

The firm's AI estate grew quickly. Different workloads needed different combinations of reasoning quality, speed, context, confidentiality, structured output, tool use and price. Some models ran locally; others were external. Some endpoints were optimised for conversation, others for embeddings, reranking, speech or batch work.

Without a common control point, each application would have needed to manage:

• credentials and provider integrations;
• model-specific request and response differences;
• local endpoint discovery;
• fallback and retry logic;
• quota, cost and latency decisions;
• upgrades and model retirement; and
• logs sufficient to understand which model actually handled a request.

That would have created duplicated code and inconsistent governance precisely as AI was becoming more important to daily work.

We introduced a proxy layer with a standard model-service interface. Applications target logical capabilities or approved aliases rather than embedding physical endpoints and provider details.

The gateway's responsibilities include:

1. Contract normalisation. Convert a common request into the form expected by the selected backend and return a predictable response shape.
2. Routing. Select an eligible model or endpoint using workload, sensitivity, quality, latency, availability and cost considerations.
3. Policy. Control which callers can use which capabilities and prevent accidental bypass of approved paths.
4. Resilience. Apply timeouts, bounded retries, fallbacks and health-aware routing without allowing retry storms.
5. Observability. Record the logical request, selected route, timing, errors and usage signals needed for operations and later evaluation.
6. Lifecycle management. Make it possible to test, introduce, redirect and retire models without forcing changes throughout the application estate.

The gateway did not decide every aspect of AI governance. Application-specific authorisation, data policy and human review remained with the systems that understood the business context. The gateway owned model access and routing, not all business decisions.

Implementation progressed from basic aggregation toward a genuine control plane:

• established a standard API shape across local and external inference;
• created logical model aliases so applications were insulated from physical deployment names;
• separated conversational generation, embedding, ranking, transcription and specialist workloads;
• introduced route groups with ordered candidates and bounded fallback behaviour;
• centralised provider credentials and endpoint configuration;
• added request metadata, correlation identifiers and usage logging;
• supported local-first, external-first and restricted-local routing patterns;
• connected retrieval systems, agent runtimes, internal applications and experimentation surfaces through the same gateway;
• created different capacity profiles for interactive periods and background ingestion windows;
• documented safe configuration changes and the need to restart, drain or switch live processes when their in-memory configuration changed; and
• tied model promotion decisions to practical evaluation rather than model reputation alone.

Model selection proved to be multi-dimensional. “Use the best model” is not an implementable policy until “best” is defined for the task.

We learned to evaluate routes against:

• Information sensitivity: whether the prompt or retrieved context may leave the controlled estate.
• Capability fit: reasoning, tool use, structured output, multilingual work, code, vision or long context.
• Service objective: interactive latency, batch throughput or background completion.
• Reliability: observed stability under the actual runtime and workload.
• Context economics: the cost and memory implications of large prompts and retrieved evidence.
• Availability: local capacity, provider health and queue state.
• Unit economics: usage price, owned-capacity utilisation and the operational cost of local serving.
• Evidence requirements: whether a workflow needs route and version information preserved for reproducibility.

This turned model choice into portfolio management. Models could be introduced for a defined role rather than promoted as universal replacements.

Retries can amplify failure

When retrieval or batch systems fan out requests, generous retries at several layers multiply traffic during an incident. Retry ownership, deadlines, circuit breaking and concurrency bounds need to be designed across the path, not independently inside every service.

Logical names need governance

Aliases reduce coupling but can obscure change. Redirecting an alias from one model to another may alter quality, safety, cost or output shape without an application deployment. Changes therefore require versioned configuration, evaluation evidence and a rollback path.

Leaders making AI architecture and partner choices should ask:

• Are applications coupled directly to one provider or model?
• Where are data-placement and fallback rules enforced?
• Can a model be replaced without changing every application?
• Is actual route selection visible after the event?
• Who owns evaluation and promotion of new models?
• How are capacity, resilience and cost managed across local and external services?

A gateway does not eliminate provider decisions. It creates the architecture and operating process through which those decisions can remain governed and changeable.