Enterprise AI Platform Engineering & Operations

Phase 0: Assess 2–6 weeks fixed fee

Most engagements start here. Before committing to a build, we inventory every AI path in the organization and map it to a named control point. The traffic coverage table is the first artifact we produce, and it is the one we check at the end of every later phase. If a path can reach a model provider without passing one of these control points, it is a finding and must be closed by egress policy.

Traffic path	Example	Primary control point	Secondary controls
API calls from internal code and automations	Scripts, n8n, internal tools, opencode run / serve	AI Gateway (virtual key required)	NGFW / ZTNA egress policy blocks direct provider endpoints
IDE / CLI coding assistants	Claude Code, opencode, other agentic CLIs	AI Gateway, via base URL set in managed client config	MDM enforces client config and MCP allow-list; NGFW / ZTNA blocks direct provider access
Enterprise AI SaaS seats	Claude Enterprise, ChatGPT Enterprise, Gemini	Vendor tenant controls (SSO, retention, DLP, connector scoping)	Admin audit API into SIEM, SWG / CASB allows only the corporate tenant
Vendor-native agents	Microsoft 365 Copilot, Salesforce Agentforce, ServiceNow Now Assist	Vendor tenant controls and contractual terms	Tenant audit logs into SIEM; procurement and license-tier selection as controls
Agents in the orchestration layer	Knowledge agent, SRE agent	AI Gateway for models and tools, policy decision point for actions	Kubernetes network policy: runtime has no direct egress to providers
Tool access (MCP) from any client	IDE assistants, agents, AI SaaS connectors	MCP gateway	MDM allow-list contains only the gateway endpoint; NGFW / ZTNA blocks direct access to vendor MCP endpoints
Unmanaged / personal AI tools in the browser	Consumer chatbots	SWG / CASB: allow for Public tier, block or warn otherwise	Existing shadow AI detection (CASB, EDR)

Beyond the traffic table, the assessment identifies unmanaged provider keys, direct model access and unapproved MCP connections. We deliver a regulatory exposure report covering EU AI Act, NIS2, GDPR and ISO 42001, and a costed technical design, layer by layer, that the customer can keep and implement with any vendor. The assessment is a fixed-fee entry point. You can keep the design and walk away.

Phase 1: Foundation — The Control Plane

Phase 1 deploys the platform scaffolding and routes all model traffic through the gateway. The platform does not deploy its own endpoint agents; it publishes rules that the customer's existing corporate controls enforce. By the end of this phase there are no direct provider keys in use and every production change is applied through Git.

MDM receives managed configurations for AI clients: gateway base URL, key retrieval method, MCP allow-list from the tool hub, approved skills and rules, and telemetry export settings. For opencode, this means a managed config directory (/etc/opencode on Linux, ai.opencode.managed plist on macOS) with enabled_providers locked to the gateway and OPENCODE_CONFIG_DIR pointing at provisioned skills. EDR receives indicators of approved AI client binaries, expected MCP processes and config file paths, and alerts on OPENCODE_PERMISSION overrides. NGFW / ZTNA receives an egress list that makes provider API domains reachable only from the gateway and agent runtime, with gateway and admin UIs published as ZTNA applications. SWG / CASB receives the approved AI SaaS tenants and Public-tier tool list. All artifacts are generated from platform registries and published as code, so a change in the platform updates enforcement through the customer's normal change process.

The platform layer itself includes an HA LiteLLM gateway with the Rust hot path, virtual keys carrying cost tags, SSO, budgets and egress rules; an OpenTelemetry collector with LLM tracing and platform dashboards; a cost data pipeline with provider billing reconciliation and weekly showback per team; and GitOps repositories, CI/CD pipelines and Argo CD for all components. Exit criteria are strict: no direct provider keys, all API traffic via gateway, weekly showback per team, no manual changes in production.

Phase 2 & 3: Governed Tools & First Agents

Phase 2 deploys the MCP Tool Hub. We install agentgateway with its registry, virtual servers and per-tool CEL policies, then connect source systems via on-behalf-of OAuth 2.1 token exchange where possible. Read-only servers come first for source code, project management, documents and Slack. Tier policy is enforced on every tool call, and no client connects directly to a system MCP server.

Phase 3 adds the agent runtime. The runtime is built on LangGraph for governed workflows with approval interrupts, backed by Temporal for durable long-running execution. Letta provides conversational agents with persistent memory and a native Slack channel. Agents run on Kubernetes in worker pools per domain or risk level — for example a general knowledge pool and a separate operations pool with prod access — which isolates noisy neighbors and limits blast radius. Each agent profile has its own identity, secrets, namespace or sandbox, and resource limits. Runtime egress is restricted to the AI Gateway and the policy decision point; model and tool access always flows through the gateway, never directly to a provider.

The first production agent is typically a knowledge Q&A agent answering in Slack from permission-aware sources. A memory pilot runs alongside it using Mem0 and pgvector, carrying source ACLs and data tier into retrieval. Skills proposed by agents are written to the skill registry as pull requests and reviewed before activation; no unreviewed skill runs in a Confidential or Restricted pool.

Phase 4: Operations Agents & Managed AIOps

Phase 4 adds write capability behind human-in-the-loop approvals, then transitions to ongoing platform operations. Write variants for Jira, CRM, ERP and Argo CD route through Slack Block Kit approvals with signed interactions and hash-chained audit. The agent workflow pauses at a LangGraph interrupt; a posted message shows the action, arguments and context with Approve and Reject buttons. Operations agents include an SRE triage agent and a CI/CD triage agent. The CI/CD agent runs on the opencode serve coding-agent pool, which is sandboxed with gVisor, uses per-session pods with warm pools, and has egress limited to the AI Gateway and Tool Hub.

FinOps becomes a full operating model in Phase 4. All cost data is normalized to the FinOps Foundation FOCUS format. Allocation keys — cost center, team, product, use case, environment — are mandatory metadata on virtual keys, agent profiles and Kubernetes labels; policy blocks creation without them. Budgets produce soft alerts and hard blocks at the gateway. Daily anomaly detection compares spend against baseline per team and agent to catch runaway loops or leaked keys. Gateway-calculated cost is reconciled against invoices, with differences over an agreed threshold investigated for unrouted traffic or price-table drift. Optimization levers include provider prompt caching, model tiering and routing, response caching keyed by clearance and session tier, batch APIs and context limits. Unit economics are reported per outcome: cost per answered question, per triaged incident, per assisted PR. Reporting cadence settles into weekly showback, monthly chargeback where used, and quarterly review of optimization and commitments.

FinOps control	Mechanism
Allocation	Mandatory keys on virtual keys, agents and labels; shared cost split by usage share
Budgets	Soft alert (for example 80%) and hard block per team, key and agent at the gateway
Anomaly detection	Daily baseline comparison per team and agent; alerts on spikes
Reconciliation	Gateway cost vs invoice; differences over agreed threshold (for example 3%) investigated
Optimization	Prompt caching, model tiering/routing, response caching, batch APIs, context limits, self-hosting after break-even analysis
Reporting	Weekly showback, monthly chargeback where used, quarterly review

Decisions We Make Together

The architecture is shaped by customer choices made during assessment and foundation. These are not optional configuration details; they define the risk posture and operating model. The full decision list from the reference architecture is addressed explicitly.

• AI model vendors and per-vendor data terms: which providers are approved and on what contractual terms.
• Content logging defaults and retention: how much prompt content to log, balancing audit needs against employee privacy and works-council consultation.
• Classification of the initial tool set: which systems are Internal, Confidential or Restricted.
• Fail-open vs fail-closed per tier: whether Public/Internal may use break-glass credentials when the gateway is unavailable.
• Production workloads: whether customer-facing production services join in a later phase.
• Self-hosted model requirement: whether Restricted data needs a customer-operated vLLM deployment.
• Showback or chargeback: whether teams see cost or are actually charged back, and which allocation keys to use.
• Observability and SIEM stack: integrate with existing tools or deploy the open-source stack.
• Existing security products: which MDM, EDR, NGFW, ZTNA and SWG products receive platform-generated rules, and who applies them.
• Source control and repository layout: GitHub, GitLab or Gitea, and mono- or multi-repo structure.
• SaaS-only MCP dependencies: accept official Slack and Atlassian MCP servers, or use self-hosted generated API tools instead.
• AI SaaS seat vendors and tenant policy baseline: audit ingestion, in-vendor DLP, connector allow-lists.
• Guardrail compute: CPU-only inline chain, or budget a GPU for asynchronous Llama Guard 4 classification.
• Workflow automation license posture: n8n fair-code, Activepieces (MIT) or Windmill (AGPL).
• EU data residency requirement: none, EU-routed providers for Confidential, or sovereign specialist / managed EU inference for Restricted.