Self-Hosted LLMs on Kubernetes (EKS)

Deployed open-source LLMs with vLLM on EKS with Karpenter autoscaling, AWQ quantization, and a custom Helm chart, exposed to internal users via FastAPI.

Context: Baxter International (Healthcare)

Problem

Teams wanted to use open-weight LLMs from Hugging Face inside the company. GPU nodes are expensive, so models had to scale with demand and fit on smaller hardware. Each new model also shouldn't need its own hand-built deployment.

Approach

vLLM on EKS. I deployed open-source LLMs with vLLM on Kubernetes (EKS).
Autoscaling. Karpenter provisions GPU nodes when models need them and removes them when they don't.
Lower compute cost. AWQ quantization of Hugging Face models lets them run on fewer or smaller GPUs.
Reusable deployments. A custom vLLM Helm chart turns deploying a new model into a configuration change.
Internal access. A FastAPI service gives internal users access to the models.

Internal users call FastAPI, which routes to vLLM servers on Karpenter-managed GPU nodes in EKS

Impact

Open-source models available to internal users through one API.
Lower compute cost through quantization and on-demand GPU nodes.
New models deployed from the Helm chart instead of from scratch.

Stack

vLLM · EKS · Karpenter · Helm · FastAPI · Hugging Face