Series · 8 parts · 47,836 tokens · ~1.7 h of reading
Building a Hybrid LLM Platform on EKS
From an empty VPC to GPU node pools, vLLM, KEDA and a cloud-vs-local inference router, built as reproducible AWS CDK you can deploy and tear down yourself.
EKSAWS CDKvLLMKEDAKarpenterHono
- 01Architecture and the Network FoundationPart 1 of a hands-on series building the EKS-based hybrid LLM platform referenced throughout this blog. We map out the full architecture, then provision the VPC, subnets, NAT, and VPC endpoints with AWS CDK — the network foundation every later part builds on.4,066 tokens · ~10 min
- 02The Control Plane, IAM, and IRSAPart 2 of our hands-on EKS series. We provision the EKS cluster into the VPC from Part 1, wire up OIDC federation and IRSA so pods authenticate without static credentials, and end with a working kubectl connection to a real cluster.3,930 tokens · ~9 min
- 03Node Groups, GPU AMIs, and the NVIDIA Device PluginPart 3 of our hands-on EKS series. We add worker nodes to the empty cluster from Part 2: a CPU system pool for add-ons and the hybrid router, a GPU pool for vLLM model servers, the NVIDIA device plugin DaemonSet, and the taints and labels that make scheduling predictable.4,133 tokens · ~10 min
- 04Platform Add-ons, the Load Balancer Controller, and KarpenterPart 4 of our hands-on EKS series. We install the two add-ons every production EKS cluster needs: the AWS Load Balancer Controller so Kubernetes Ingress objects provision real ALBs, and Karpenter for cost-aware autoscaling — including the GPU NodePool that scales to zero between inference workloads.6,960 tokens · ~13 min
- 05Serving Local Models with vLLM and KEDAPart 5 of our hands-on EKS series. We deploy vLLM model servers on the GPU pool from Part 4, load Qwen2.5-7B model weights from Amazon S3 via an init container, and wire KEDA autoscaling that scales replicas with live queue depth and drives GPU nodes to zero overnight.7,167 tokens · ~16 min
- 06The Hybrid RouterPart 6 of our hands-on EKS series. We build a TypeScript/Hono router that sits in front of both vLLM and the Anthropic API, routes each request to the right backend based on model name and complexity heuristics, and falls back to cloud when the local model is cold-starting.7,495 tokens · ~16 min
- 07Observability and Cost TelemetryPart 7 of our hands-on EKS series. We instrument the TypeScript router with OpenTelemetry, upgrade Prometheus to kube-prometheus-stack for GPU and vLLM metrics, add Grafana Tempo for distributed traces, and wire Langfuse so every request shows its backend, token count, and dollar cost.7,204 tokens · ~14 min
- 08Testing, Load, and ExamplesThe final part of our EKS series. We write integration tests with Vitest, load-test the ALB with k6, build three real-world TypeScript workloads that prove the hybrid routing works, and use the Grafana and Langfuse dashboards from Part 7 to verify the platform under traffic.6,881 tokens · ~16 min