Global Advisors built and operated a local inference capability across several generations of compute. The journey began with modest experimentation and developed into a heterogeneous platform serving large language models, embeddings, reranking, speech transcription, retrieval workloads and agent tools.

The work created direct experience of the difference between demonstrating a model and running inference as an internal service: hardware selection, memory limits, runtime compatibility, numeric stability, multi-accelerator communication, workload placement, capacity management, observability and operational recovery all became part of the design.

Maturity: operational platform capability, with continuing benchmarking, capacity planning and hardening across workload classes.

That required answering practical questions:

• Which workloads genuinely benefit from local execution?
• How much usable capacity is created by a given memory and interconnect profile?
• Which runtime paths remain viable on different accelerator generations?
• How should interactive requests, batch retrieval and speech processing be separated?
• What operational burden is justified by sovereignty or cost advantages?

The platform evolved through successive stages rather than one procurement decision.

Early low-capability hardware established a blunt but valuable lesson: possessing accelerators is not the same as having useful modern inference capacity. A later workstation-class system made local model execution materially useful and exposed the trade-offs between model size, precision, context length and memory.

Dedicated retrieval compute then began separating background knowledge-processing work from ad hoc user interaction. Enterprise accelerator capacity enabled much larger models and long-context serving. A further expansion used a different accelerator class to add cost-effective capacity for general inference, embeddings, reranking and transcription.

The resulting architecture was intentionally heterogeneous. Different hardware generations, memory profiles and interconnects were assigned roles rather than forced into one homogeneous cluster abstraction.

We separated the estate into workload classes:

• Large-model serving: memory-intensive models distributed across tightly connected accelerators.
• General interactive inference: smaller and mid-sized models optimised for responsive day-to-day use.
• Retrieval support: dedicated embedding and reranking workers with bounded concurrency.
• Speech processing: transcription and speaker-separation workloads operated as a first-class knowledge-ingestion path.
• Experimentation: lower-friction environments where models and quantisation approaches could be evaluated before promotion.

We also avoided assuming that one serving runtime would be best for every combination. Runtime selection depended on hardware support, model architecture, precision path, context requirements, batching behaviour and operational stability.

The implementation work spanned physical infrastructure and application integration:

• commissioned multiple generations of local accelerator systems;
• designed power, cooling and staged bring-up for high-density compute;
• analysed device placement, address-space pressure and processor-to-device locality;
• configured multi-accelerator tensor parallelism and peer communication;
• built reproducible containerised serving environments;
• aligned drivers, compute libraries, framework versions and runtime builds;
• created persistent model and compilation caches to reduce recovery time;
• tuned precision, cache allocation, block sizes and session length for long-context use;
• deployed separate workers for embeddings, reranking and speech processing;
• connected the serving layer to a model gateway, retrieval platform and agent workflows;
• implemented health checks, logs and repeat-request tests; and
• captured working compatibility patterns so a future upgrade would not erase hard-won knowledge.