Lead Inference Engineer

About Aleroy

Aleroy is building a unified platform for high-performance AI inference, making it easier for companies to deploy and run AI models at scale.
Through a single API, Aleroy provides access to ready-to-use and custom AI models across cloud infrastructure, dedicated capacity, and customers' own hardware. Our platform optimizes performance, cost, and deployment flexibility while giving enterprises control over where and how their AI workloads run.
We work in small teams, collaborate directly, and focus on solving complex engineering challenges to build reliable, scalable AI infrastructure.

About the Role

You will work across inference serving, GPU performance, distributed systems, benchmarking, and model optimisation to make our serving infrastructure faster and more efficient while maintaining a clear quality bar.
This is the senior technical seat for AI serving at Aleroy. You will set the serving and measurement roadmap, establish engineering standards, and guide engineers working across serving, metrics, and testing.
The role is highly hands-on. You’ll be expected to work directly with the inference stack, identify performance bottlenecks, implement improvements, and demonstrate their impact through rigorous and reproducible measurement.

What You Will Own

• Inference Serving: Own and improve the FAR Labs inference serving stack, including prefill/decode disaggregation, continuous batching, KV-cache management, cache-aware routing, speculative decoding, and other serving optimisations.

• Serving Performance: Improve throughput, latency, memory efficiency, and serving cost while maintaining clearly defined model-quality standards.

• GPU Performance Engineering: Identify and eliminate GPU performance bottlenecks across the inference stack using CUDA, Triton, profiling, kernel optimisation, memory optimisation, and appropriate precision strategies.

• Mixture-of-Experts Serving: Drive efficient serving of large MoE models, including expert parallelism, all-to-all communication, expert-load balancing, and distributed execution.

• Quantization & Model Optimisation: Implement and evaluate quantization and other model optimisation techniques while measuring accuracy against appropriate full-precision references.

• Benchmarking & Measurement: Own rigorous, reproducible benchmarking across serving performance, quality, and cost. Establish measurement methodologies that can withstand external technical scrutiny.

• Performance Instrumentation: Build and improve the instrumentation required to understand system behaviour, identify bottlenecks, compare configurations, and validate improvements.

• Distributed Serving: Make technical decisions around routing, scheduling, caching, and workload distribution across diverse hardware environments.

• Technical Roadmap: Define the AI serving and measurement roadmap and determine which performance problems and infrastructure improvements should be prioritised.

• Engineering Standards: Establish clear technical standards around inference performance, benchmarking, reproducibility, testing, and production readiness.

• Technical Leadership:Guide and mentor engineers working across serving, metrics, and testing while remaining deeply involved in implementation and technical problem solving.

About You

• Strong hands-on experience building, optimising, or operating LLM inference-serving systems in production.

• Deep understanding of inference stacks such as vLLM, SGLang, TensorRT-LLM, or similar technologies.

• Strong understanding of paged attention, continuous batching, disaggregated serving, KV-cache management, and modern inference architectures.

• Strong GPU performance engineering experience using CUDA or Triton.

• Strong understanding of GPU performance characteristics including memory bandwidth, compute utilisation, model bandwidth utilisation, FLOPs utilisation, and modern GPU precisions such as FP8 and FP4.

• Experience serving large Mixture-of-Experts (MoE) models, including expert parallelism, all-to-all communication, and expert-load balancing.

• Hands-on experience with model quantization and measuring quality against full-precision references.

• Strong benchmarking experience, including load generation, latency percentiles, throughput-at-SLO, quality measurement, and reproducibility.

• Strong understanding of distributed systems, particularly routing and scheduling workloads across varied hardware.

• Ability to analyse complex performance bottlenecks across models, serving infrastructure, networking, and hardware.

• Demonstrated technical leadership experience leading an inference, serving, ML infrastructure, or similarly complex engineering initiative.

• Ability to set technical direction while remaining hands-on with implementation and performance optimisation.

• Experience with speculative decoding and understanding its behaviour under production load is a strong plus.

• Open-source contributions to inference or serving frameworks are a strong plus.

• Experience serving reasoning models, long-context models, or agentic multi-turn workloads is a plus.

• Experience with energy-aware serving, including throughput-per-watt or power-constrained environments, is a plus.

• Experience with confidential computing or data-residency-aware serving is a plus.

• Production experience with model compression techniques such as structured sparsity, activation sparsity, or KV compression is a plus.

• English level: Advanced (C1).

What We Offer

• Real ownership of the backend that powers Aleroy inference platform

• Influence over backend architecture and product direction

• Hands-on work on hard problems in distributed systems, real-time streaming, and LLM serving

• A small, senior team with fast execution and low bureaucracy

• Competitive salary with performance-based incentives

• 24 days annual leave plus public holidays

• Health insurance

• Modern office in Yas Creative Hub