Skip to content
Back to projects
BaxterDec 2023

LLM API Key Management Platform

Built the backend that controlled access to 15+ internally hosted and Bedrock LLMs, which grew into an internal developer platform used daily by 150+ developers.

DevOpsLLM
Developers request API keys that grant access to self-hosted and Bedrock models

Context: Baxter International (Healthcare)

Problem

As the company started experimenting with LLMs, it hosted more and more models: open-weight models from Hugging Face, and later models on Amazon Bedrock. With more than 15 models available at a time, developers needed a controlled way to get access, and the company needed to know who was using what.

Approach

  • API key backend. I built a FastAPI backend for issuing, validating, and revoking LLM API keys, with keys stored in RDS.
  • Company authentication. Developers sign in with the company identity provider (Okta), so keys are tied to real users.
  • Notifications. SMTP email notifications keep developers informed about their keys.
  • IDE integration. The backend integrates with a Visual Studio plugin, so developers can use the models from their editor.
  • From self-hosted to Bedrock. It started with the models we hosted ourselves and later added Bedrock models for internal use.

Developers authenticate, receive a key, and call self-hosted or Bedrock models through the platform

Impact

  • Used daily by 150+ developers across the company.
  • Grew from an API key service into an internal developer platform.
  • When the company later adopted a LiteLLM gateway for key management, the platform was repurposed for other developer tooling.

Stack

Python · FastAPI · SQLAlchemy · Pydantic · AWS RDS · Okta · SMTP