The Governed AI Gateway & MCP Tool Hub

The AI Gateway

The AI Gateway is a single OpenAI-compatible endpoint that fronts every model the enterprise uses: commercial APIs, open-weight providers and self-hosted inference. Provider swaps become configuration changes rather than code rewrites, and no application or team holds a raw provider key. Reference implementation is LiteLLM running its Rust hot path in production. Where SSO, RBAC at scale and full audit logs must remain in the free tier, the gateway can be fronted with Kong AI Gateway 2.0.

Authorization is enforced per virtual key. Each key is bound to an authenticated principal — a user, a workload, or a delegated user+agent identity — and carries its own allowed model list, allowed tool set, data-tier clearance, rate limits (RPM/TPM) and budget. Keys expire automatically and rotate without reconfiguring clients. The gateway consults the policy decision point on every call, so tier checks and custom rules are not bolted on afterwards but evaluated inline before any provider is reached.

Reliability is built in: retry with backoff, cross-provider fallback and load balancing. A fallback target must satisfy the same clearance tier and residency attribute as the primary target, so resilience never becomes a data-leak path. Availability is treated seriously because the gateway is a dependency for all AI usage. It runs at least three replicas across zones, stores state in a managed PostgreSQL HA pair and keeps rate-limit counters in Redis. The gateway is exposed only through the corporate ZTNA or internal network; for Confidential and Restricted tiers it fails closed. Break-glass credentials for Public and Internal use are time-boxed, individually issued and fully audited. In production, the admin UI is read-only; every real change flows through Git.

Capability	Mechanism
Authentication	SSO for admin UI; virtual keys bound to IdP identity with short expiry and automatic rotation
Authorization	Per key: allowed models, allowed tools, clearance tier, RPM/TPM, budget
Reliability	Retry/backoff, cross-provider fallback, load balancing; fallback must match tier and residency
Residency routing	Each model deployment carries a residency attribute; routing and fallback respect it
Availability	3+ replicas across zones; managed PostgreSQL HA; Redis rate counters; private ZTNA endpoint; fail-closed for Confidential/Restricted
Break glass	Time-boxed credentials for Public/Internal, individually issued, every use alerted and audited
Configuration	All config in Git, applied by Argo CD; admin UI read-only in production

Inline Guardrails & DLP

Every request and response passes through a guardrail chain designed to enforce policy without adding unacceptable latency for interactive use. The inline input chain runs on CPU and acts as a fast gate: Llama Prompt Guard 2 classifies prompt-injection attempts in under 20 ms; Presidio detects and masks PII, credentials and customer identifiers before they reach a model or are stored; NeMo Guardrails enforces topic and workflow constraints; and OPA evaluates the data-tier check against the policy decision point. Flagged or sampled traffic is then classified asynchronously by Llama Guard 4 where a GPU is budgeted.

The output side is just as important. Tool outputs and model responses are not simply sanitized and forwarded. The chain is: schema validation first, then content and DLP policy, then output filtering, then provenance and trust tagging, and only then the result is passed to the model or user. This ordering matters because it prevents a downstream detector from being bypassed by an earlier transformation. Provenance tags separate tool data from system instructions in the agent prompt, which is one layer of the prompt-injection defence.

On prompt injection specifically, the platform is honest about residual risk. There is no complete mitigation for prompt injection. The controls reduce how often an injection succeeds and limit the blast radius when one does: tool outputs pass guardrails, results are tagged as data, each agent carries a minimal allow-listed tool set, egress is restricted to block exfiltration, and SIEM rules watch for unusual tool sequences. Because risk remains, high-impact tools stay behind human approval even when all automated controls are enabled.

The MCP Tool Hub

The Tool Hub is the single door to enterprise systems. Source code, project management, documents, CRM, ERP, delivery, observability and data platforms are exposed as MCP tools through one authenticated Streamable HTTP endpoint. Agents and IDE assistants never hold system credentials, and no client connects directly to a backend MCP server. Reference implementation is agentgateway from the Linux Foundation; it authenticates callers via OAuth 2.1, authorizes each tool call with per-tool CEL policies checked against the PDP, exchanges or injects downstream credentials, applies rate limits and output guardrails, and emits OpenTelemetry spans and audit logs.

The architecture has four layers. The MCP gateway is the data plane. Virtual servers sit above it: curated toolsets per role or agent, such as "engineering-read" or "sre-operations", which keep the number of tools per session small and improve selection accuracy while lowering token cost. Backend servers run centrally on Kubernetes, one deployment per system and scope variant, with vendor-hosted remote MCP servers proxied through the gateway rather than called directly. The registry is the control plane: one entry per server capturing system, owner, data tier, scope, auth pattern, version, pinned tool-definition hash, approval status and cost tags. That registry feeds gateway config, MDM allow-lists, policies and FinOps tags.

Downstream authentication follows four patterns, chosen per system. On-behalf-of token exchange is preferred for systems federated with the IdP, such as source code, documents and project management. Per-user OAuth brokered by the gateway is used for SaaS without token exchange, common for CRM and many project tools. Scoped service accounts are reserved for ERP and similar systems without per-user API auth, always combined with task-level tools and approvals for writes. Workload identity is used for cloud and Kubernetes APIs, where the server's service account binds to a project-scoped token. Every request carries an authenticated principal: a user, a workload, or a delegated user+agent identity; unattended agents run under workload identity, not a user's identity.

Auth pattern	How it works	Use for
On-behalf-of token exchange	Gateway exchanges the user's IdP token for a system token scoped to that user	Code, documents, project management when federated with IdP
Per-user OAuth, brokered	User consents once; gateway stores refresh tokens in the secrets vault and injects them	SaaS without token exchange, such as many CRM tools
Scoped service account	Technical user with minimal permissions; gateway restricts tools and arguments per caller	ERP and systems without per-user API auth
Workload identity	Cloud or Kubernetes identity bound to the server's service account	Argo CD, Kubernetes, cloud APIs

MCP servers enter the catalog in order of preference. Vendor official servers are used when available and maintained; maintained open-source servers are forked, pinned to a reviewed version and built internally; systems with OpenAPI or OData specs generate a filtered, read-only-by-default tool set; and custom task-level tools built with FastMCP cover ERP business actions and cross-system tasks. The design rule is task-level tools over raw API mirrors: a small set of well-defined business actions is easier to authorize, test and audit than hundreds of endpoints.

Tool Poisoning & Supply Chain Defense

The "rug pull" threat — a silent change to a tool's name, description or schema that tricks a model into calling the wrong action — is blocked by treating the tool registry as immutable code. The registry stores a hash of each tool's name, description and schema. The gateway rejects any definition whose hash does not match the approved entry, so a compromised package cannot alter tool semantics without a reviewed pull request. Before a definition reaches the registry, a CI pipeline scans it in three stages: mcp-scan from OWASP as the fast gate, NVIDIA SkillSpector for skill packages, and AI-Infra-Guard for MCP servers.

Runtime isolation limits blast radius. Each backend server runs in its own namespace, as a non-root container, with a read-only filesystem and a network policy that allows egress only to its target system. Credentials are scoped to that system. The default variant is read-only; write variants are separate servers with their own identity and are granted only to the virtual servers that need them. Argument-level PDP checks enforce allowed Argo CD projects, repositories, SQL templates, record types and amount limits for ERP actions. Output size limits, PII masking and tier tagging prevent leakage through oversized or over-privileged results. Rate limits protect backend systems, and every call is audited with user, agent, virtual server, tool, argument hash, result size, policy decision and approval ID.

Tool design rules reinforce this. Virtual servers contain tens of tools, not hundreds; very large APIs use a search-and-execute pair only when argument-level policy checks are in place. Naming follows system_object_action, pagination and result limits apply to every list tool, and every tool has an owner, a tier, a read/write flag and test cases in the agent evaluation suite. Quarterly re-certification retires unused tools before they become forgotten attack surface.