RBAC and API-Key Sprawl in the Age of LLMs
Ask a platform team six months into their AI rollout where their model credentials live, and the honest answer is usually “everywhere.” A provider key in a data scientist’s notebook. Another in a CI secret. A shared key in a Slack message from the early prototype days that three services still depend on. A fourth pasted into an environment variable on a VM nobody remembers provisioning. Each one grants full, unscoped, unattributed access to a paid API — and no single system knows how many exist or who is using them.
This is API-key sprawl, and it is the quiet security and cost problem underneath most enterprise LLM programs. It doesn’t cause an outage, so it rarely gets prioritized, right up until a key leaks, a bill spikes, or an auditor asks a question nobody can answer. For engineering leaders, getting ahead of it is far cheaper than cleaning it up. This piece lays out why the classic model breaks down and what a durable access-control architecture looks like when dozens of teams share model access across multiple providers.
Why provider keys don’t scale across an organization
A raw provider API key has a set of properties that are fine for one developer and dangerous for an enterprise.
It’s a bearer credential with no identity. Whoever holds the key is the caller. Provider-side logs show that “the key” made a request, not that Priya on the fraud team made it. When something goes wrong, you can’t attribute it to a person, a team, or an application without external bookkeeping you probably don’t have.
It’s all-or-nothing. A provider key typically grants access to every model and capability on that account. There’s no built-in way to say “this team may use the cheap model but not the expensive one,” or “this application may read embeddings but not run image generation.”
It multiplies per provider. The moment you adopt a second or third model provider — which, for reliability and cost reasons, most serious teams do — every team now juggles a separate key per provider, each with its own rotation story and its own blast radius.
It rotates badly. Because keys get copied into many places, rotating one is a scavenger hunt. So teams don’t rotate, and long-lived secrets accumulate.
The result is a credential estate that is impossible to inventory, impossible to attribute, and impossible to govern with least privilege. That’s not a tooling failure by any one team; it’s the predictable outcome of using raw provider keys as an organizational access model.
The shift: identity-based access at a central control point
The way out is to stop handing provider keys to humans and applications at all, and instead route all model traffic through a single control point that authenticates callers with your own identities and enforces access policy centrally. This is one of the core reasons organizations put an AI gateway between their applications and their providers: the provider keys are held once, by the gateway, and never distributed. Callers authenticate to the gateway, the gateway decides what they’re allowed to do, and only then does it talk to the provider on their behalf.
That single move changes the security posture in three ways. Provider keys stop leaking into notebooks and CI because nobody but the gateway ever holds them. Every request now carries a real identity — a user, a team, or an application — so attribution is automatic. And access becomes a policy you set centrally rather than a property of whoever happened to receive a key.
Two kinds of tokens: humans versus workloads
A practical access model distinguishes between people and running software, because they have different lifecycles and risk profiles.
Personal tokens are tied to an individual user and are the right choice for development and experimentation. They inherit the person’s permissions, they’re easy to trace back to a human, and they can be revoked when that person changes roles or leaves.
Workload or service tokens are tied to a virtual identity rather than a person, and are the right choice for production applications. A production service shouldn’t authenticate as “Priya” — it should authenticate as “the fraud-scoring service,” with its own scoped permissions and its own rotation schedule, independent of any employee. TrueFoundry’s AI Gateway, for example, distinguishes personal access tokens for developers from virtual-account tokens for production workloads for exactly this reason.
Getting this separation right means a departing employee’s revoked access never takes down a production service, and a compromised development token never has production-level reach.
Designing the permission model
With identity established, the next question is what each identity may do. A workable model grants access at the level of model accounts — logical groupings of provider models — and assigns roles that separate using a model from managing it.
A user role can call the models in a group but can’t change settings or alter who else has access. A manager role can add or remove models and administer permissions. TrueFoundry’s access control model follows this shape: permissions are assigned to teams or individuals at the model-account level, so granting the analytics team access to a set of models is one action, not a key-distribution exercise. Assigning the permission to a team means every member inherits it, and removing someone from the team removes their access — no orphaned keys left behind.
The design principles that make this durable are familiar from the rest of your security program, now applied to models: default to least privilege so identities start with no model access and are granted only what they need; grant to groups rather than individuals so access tracks team membership; and separate use from administration so the people who call models aren’t automatically the people who can change who else can.
Cost control is an access-control problem too
Key sprawl isn’t only a security issue; it’s a financial one. When credentials are unattributed, so is spend — you get one big provider bill and no way to allocate it. When access runs through an identity-aware control point, spend can be attributed to the same users, teams, and applications, and you can attach budgets and rate limits to those identities. A runaway agent stuck in a loop or a buggy batch job hits a per-team ceiling instead of the whole organization’s. That turns “why did the bill triple?” from a forensic investigation into a dashboard query.
A migration path
Most organizations can’t flip a switch, but the sequence is straightforward. Stand up a central gateway and move the provider keys into it, so there’s one home for the real secrets. Point new projects at the gateway from day one using personal tokens for development and workload tokens for anything heading to production. Migrate existing services off their embedded provider keys one at a time, revoking each raw key as its last consumer moves over. Then define your role and group model so access maps to your org chart, and layer budgets and rate limits onto the same identities.
None of these steps is dramatic, and each one shrinks the credential estate you’re exposed to.
The takeaway
Raw provider API keys are a fine credential for one developer and a liability for an enterprise: they carry no identity, grant blanket access, multiply per provider, and resist rotation. Routing model traffic through an identity-aware control point replaces that sprawl with something you can actually govern — real identities on every request, least-privilege access granted to teams, a clean split between using and managing models, and spend attributed to the people and applications responsible for it. The organizations that treat model access as a first-class part of their identity and access strategy, rather than a pile of secrets to manage, are the ones that scale AI across many teams without losing control of who can do what.
