Platform reference

Technical documentation

Deployment models, runtime components, AI capabilities, workflow orchestration, and data interoperability.

Technical reference for the Fontana platform: deployment models, AI capabilities, workflow orchestration, and data ingress/egress. The production architecture is designed for SOC 2 diligence - cluster-per-tenant segregation, immutable releases, and audit-ready logging and strong security controls.

What Fontana is

Fontana is an Agent Harness, Knowledge Graph, Workflow Engine, and AI data analysis platform. Teams design visual pipelines on a canvas - ingest data from files, APIs, or connectors; transform and validate with operations and AI; analyze results via visual reports and data queries; and export or push outputs to downstream systems. Configurable agents, tools, and a searchable knowledge base help users build, run, and govern data workflows without writing application code.

Core runtime components

Component Technology Role
Host AWS / Azure / GCP / on-prem Deploy anywhere - platform-agnostic infrastructure
Container orchestration Kubernetes (K3s) Multi-tenant container orchestration with per-workspace cluster segregation
Flow App React (PWA SPA) Web UI - canvas, grids, AI chat, admin panels
Workflow engine Node.js, Python Server-side execution of workflow graphs
Real-time backend Convex Application backend - config, AI orchestration, persistence
Auth Zitadel Self-hosted OIDC identity per workspace - SSO, MFA, SCIM provisioning
Secrets storage HashiCorp Vault Per-workspace encrypted secret storage (API keys, connector credentials) - SOC 2-aligned controls
Observability HyperDX Platform-cluster OTLP collector and operator dashboards (shared across workspaces)
AI code sandbox OpenSandbox Isolated code execution for agents via MCP
Data replication Airbyte (PyAirbyte) Data ingress and egress - catalog connectors in the workflow engine

Deployment: deploy anywhere

Fontana runs on Kubernetes. Production images are published to GHCR (GitHub Container Registry) and pulled to your environment as pinned, immutable releases. The same stack deploys to major clouds and on-premises - including customer VPC patterns for regulated workloads and SOC 2-aligned deployments.

Deployment models

Model Notes
Managed cloud Fontana operates the infrastructure: host provisioning, Kubernetes, pinned release upgrades, backups, scaling, and monitoring. Preferred Deployment Option
Customer VPC You operate the cloud account and VPC in AWS, Azure, or GCP. Fontana supplies pinned OCI images and install bundle from GHCR; you manage networking, backups, scaling, and the data residency boundary under your cloud contract.
On-premises You operate Kubernetes on your hardware or private cloud. Pull pinned OCI images from GHCR or mirror them to a private registry; you own facility security, capacity, backups, and change control on your network.
Dedicated host One physical or virtual host per customer (no shared multi-tenant box). Available Fontana-managed or customer-operated; same cluster-per-workspace model with a smaller blast radius when hard isolation is required.

Architecture diagram

Platform cluster hosts the welcome gateway and shared stateless services (HyperDX/OTLP, Docling, OpenSandbox). Each workspace cluster is a separate k3d environment with Traefik ingress, Flow, Platform API, workflow engine, Convex, Postgres, Vault, Zitadel, and Gatus - no shared database or secret store across tenants.

Multi-tenancy

Multiple workspaces can share one host, each in a separate Kubernetes cluster with separate data, Vault, and identity. This cluster-per-tenant model is the platform’s primary SOC 2 segregation boundary. Dedicated hosts are available when hard isolation is required.

Upgrades and rollback

  • Workspace upgrades take an automatic snapshot before applying changes - an auditable change-control record on the host.
  • Rollback restores both data and the pinned release from a snapshot.
  • Production deploys use immutable image tags for reproducibility, supply-chain integrity, and audit.

Identity and access

Each workspace runs a self-hosted Zitadel instance - the SOC 2-aligned identity plane for that tenant. Zitadel supplies authentication, federation, and MFA; Flow and Convex enforce application-level RBAC on top.

Capability Provider Implementation
Single sign-on (SSO) Zitadel Per-workspace identity plane. Federate corporate IdPs so users sign in with organisational credentials.
OIDC Zitadel Authorization Code + PKCE for the Flow SPA. Issuer, client, and redirect URIs are provisioned per workspace.
SAML federation Zitadel Upstream SAML identity providers can be linked for enterprise SSO alongside OIDC clients.
Multi-factor authentication (MFA) Zitadel TOTP and WebAuthn (security keys / passkeys) enforced through Zitadel login policy.
SCIM provisioning Zitadel Inbound SCIM v2 for automated user provisioning and deprovisioning. Directory groups map to workspace access.
Token introspection Zitadel RFC 7662 introspection of opaque access tokens for platform-api and workflow-engine - backend services never trust browser tokens blindly.
Application RBAC Flow / Convex Fine-grained permissions on workflows, admin surfaces, and platform APIs via `@fontana/rbac` - roles, teams, and scoped grants.
Role assignment Zitadel → Flow Coarse roles are assigned in Zitadel (including SCIM-provisioned users); Flow stores the effective role on the user record. No in-app privilege escalation on the Users table.
Service authentication Zitadel Dedicated OAuth clients for workflow-engine, platform-api, and automation jobs - credentials rotated via Vault-backed provisioning.
Session lifecycle Zitadel OIDC Standard OIDC logout and token refresh. Flow fails closed when discovery or introspection is misconfigured.

AI capabilities

Fontana embeds AI across workflow design, data transformation, and in-app chat. All LLM calls go through a gateway abstraction with Bring Your Own Key (BYOK) - your API keys stay in per-workspace Vault (SOC 2-aligned secret storage). Model use is approval-gated: administrators configure approved LLMs for the organisation. BYOK keys can be set at Person, Team, or Org level for fine-grained logging and cost control. The platform supports 300+ models via 12+ approved gateways.

Multi-gateway routing

Gateways only appear when their API key is configured at a matching scope. Unconfigured gateways are disabled in the selector. Embeddings for Knowledge Graph search use the same gateway stack with configurable gateway priority (OpenRouter → OpenAI → LLM API).

Approved gateways

Supported model providers

Zero Data Retention (ZDR)

Fontana supports ZDR on every gateway where the upstream provider exposes it natively - via your BYOK contract and, for unified gateways, the provider's own routing and policy controls. Retention posture follows the gateway you route through and the key you configure.

OpenRouter ZDR enforcement

When chat routes through OpenRouter, ZDR is governed by OpenRouter's ZDR policy on your account. OpenRouter tracks per-endpoint data policies (retention and training) and can restrict routing to ZDR-only endpoints at account, guardrail, or request scope. Fontana traffic uses your OpenRouter API key, so account-wide and guardrail ZDR settings apply to all calls through that key.

Model group (OpenRouter) Effect when ZDR is enabled for that group
Anthropic Removes first-party Anthropic endpoints (AWS Bedrock and Google Vertex routes remain available)
OpenAI Removes first-party OpenAI endpoints (Azure routes remain available)
Google Removes AI Studio endpoints (Vertex routes remain available)
Non-frontier Removes all other non-ZDR endpoints
  • Account-level: each model-group toggle in OpenRouter privacy settings restricts all requests on that account to ZDR endpoints for that group.
  • Guardrail-level: separate fields per group - enforce_zdr_anthropic, enforce_zdr_openai, enforce_zdr_google, enforce_zdr_other - so different API keys or org members can carry different ZDR posture.
  • Per-request: provider: { zdr: true } in the API body ORs with account and guardrail settings (it can only tighten routing, not override ZDR off when account policy requires it). Useful for one-off ZDR-only calls without a global toggle.
  • Endpoint list: ZDR-capable endpoints are published at openrouter.ai/api/v1/endpoints/zdr and updated when provider policies change. OpenRouter treats unknown endpoint policies conservatively (assumed retain + train).
  • Caching: in-memory prompt caching on the provider side is not classified as retention for ZDR routing.
  • OpenRouter itself: OpenRouter does not retain your prompts unless you opt in to prompt logging.

Direct gateways and ZDR

When you route through a direct gateway (not OpenRouter), ZDR and no-training posture comes from that provider's API and your enterprise agreement:

  • Anthropic, OpenAI, Google, Mistral, AWS Bedrock - direct API calls with your keys; zero-retention and no-training modes depend on the product tier and account settings you hold with that vendor.
  • LLM API and other OpenAI-compatible hosts - follow the host's published data policy for the key you configure.
  • Operational control in Fontana - admins approve which gateway and model IDs are reachable in production; combined with BYOK, teams can standardise on direct enterprise endpoints or OpenRouter ZDR-only routing without shadow usage.

Provider vs platform logging: upstream LLM providers apply ZDR and retention under your BYOK contract and gateway settings (above). Separately, Fontana records token usage, latency, and routing metadata for cost and compliance. Optional in-platform AI audit capture (admin audit logs, ai_tool_calls records, and permission-gated LLM debug) can retain prompt and completion excerpts inside your deployment; scope those surfaces to investigators who need them, or rely on metadata-only logging when message bodies must not be stored.

In-app agents

Agents are configurable AI personas for Flow chat. Definitions follow the open AGENTS.md pattern: YAML frontmatter for structured config plus a Markdown body for the system prompt. In Admin → Agents, each agent can be edited as a structured form or as one raw document; both views serialize to the same format used for seed files (for example packages/convex-seed/src/agents_data/). Shipped defaults include Workflow Architect (general assistant) and Canvas agent (workflow graph editing).

Per-agent configuration includes:

Setting Description
Id and display name Stable identity in YAML frontmatter
Description and system prompt Short summary plus Markdown instructions (document body)
Language model Approved model override; empty uses deployment default
Tools Allowlisted entries from the audited tool catalog
Scopes Flow chat vs internal programmatic calls
Sub-agents Delegation targets that run work in child threads
Handoff agents Same-thread transfer targets when the task surface changes
Skills Assigned instruction packages; see Skills below
Knowledge Graph namespaces Retrieval boundaries for regulated corpora
Vector source Keywords for agent embedding and routing
Call settings Optional JSON for temperature, topP, maxOutputTokens, and related LLM parameters
Reasoning effort Agent default; per-thread chat controls can override
Context window slots Agent default; per-thread chat controls can override
KB injection score threshold Agent default; per-thread chat controls can override

Skills

Fontana uses agentskills.io skill definition standard for describing agent skills.

Each skill is a versioned SKILL.md with YAML frontmatter and a Markdown procedure body. Fontana validates packages at create, edit, and import. Supported frontmatter keys:

Frontmatter key Required Description
name Yes Human-readable skill title shown in the catalog and admin UI
description Yes When the model should consider activating this skill; injected into agent context before activation
allowed-tools No YAML list of tool ids from the audited catalog; unioned into the agent's effective tools while the skill session is active

The Markdown body after the frontmatter holds the full procedure (steps, guardrails, examples). It is not sent on every turn; see progressive disclosure below.

Permissioning: skills are assigned per agent in Admin → Agents. Assignment means the skill may be used in that agent's threads, not that its full text is always loaded. Unassigned skills are invisible to the model.

Progressive disclosure: assigned skill descriptions are auto-injected into agent context as a lightweight ## Available Skills catalog on each turn so the model can choose whether to activate one. Full skill content (the complete SKILL.md body) is auto-injected only after the model calls activate_skill with the skill's stable id. While a skill session is open, bundled references are discoverable via fetch_skill_resource tool calls (manifest-bound paths; not bulk-inlined). The model closes a session with close_skill; closed skill internals are redacted from later provider context while the transcript retains audit rows.

Knowledge Graph

Knowledge Graph

Build a knowledge graph to model your firm's standards, rules, and decisions.

Documents live in Convex (knowledge_graph_sources) with namespace scoping, dot-path folders, version history, and optional document-graph edges for navigation. Namespace boundaries enforce least-privilege retrieval for regulated corpora.

Knowledge Graph documents are indexed and embedded as 1536-dimensional vectors (text-embedding-3-small). Before each turn, Fontana agents automatically run vector search over their permitted namespaces and inject full document text for hits at or above the agent's kbInjectionScoreThreshold (minimum similarity score; default 0.35).

Embeddings and retrieval

  • Model: text-embedding-3-small - 1536-dimensional vectors stored on each document and indexed for semantic search scoped by namespace
  • Embedding input: vectorSource, title, and description only - the full document body (text) is never sent to the embedding provider
  • Descriptions: Knowledge Graph descriptions allow optimised indexing and embedding generation, improving vector search results.
  • Query path: agent/user queries are embedded through the same BYOK gateway stack (priority: OpenRouter → OpenAI → LLM API); vector search returns slug and description - agents fetch full text via @slug direct lookup
  • Namespace scope: each agent’s knowledgeGraphNamespaces restrict which corpora are searchable

Knowledge Graph Document content types

Structured documents use CSV, TSV, or JSON dataTypes. Unstructured prose and Markdown (YAML frontmatter) use text. PSV and other delimited formats are supported on workflow File Input - see Ingress below.

Knowledge Graph namespaces

Knowledge Graph documents are organised in three levels: a namespace (top-level corpus boundary), a dot-path folder within that namespace, and individual documents addressed by slug. Namespaces and folders are managed in the Knowledge Graph admin UI; each document carries version history.

Each agent is permissioned to a collection of namespaces via knowledgeGraphNamespaces in Admin → Agents. Retrieval, embedding search, and @slug lookup stay inside those namespaces. That keeps regulated corpora segregated and narrows what each agent can see, which improves relevance and keeps agent behaviour focused on the right domain.

Tools and MCP

Agents invoke capabilities through a tool catalog - structured functions the model can call server-side, with JSON arguments in and structured results out. Fontana combines native internal tools (workflow, data, canvas, documents) with MCP connections to external systems. Both paths share the same allowlist, audit trail, and messaging runtime; MCP is the integration style for third-party tool servers, not a substitute for built-in workflow or Agent Interop.

MCP

Use MCP to access external data sources, perform financial data lookups, or communicate directly with your firm's tooling.

  • Built-in and MCP tools are assigned per agent; wildcards (mcp_<slug>.*) expand against cached snapshots
  • MCP discovery refresh is manual from Tools admin; local stdio MCP is not supported from Convex actions
  • All tool executions write to ai_tool_calls metadata for compliance review

OpenSandbox (AI sandboxing)

  • Shared platform installation exposes an MCP endpoint over HTTP
  • Per-workspace bearer tokens for attribution and isolation
  • Stateless, TTL-expired sandboxes for command and file operations

Agent interop

Fontana supports A2A, ACP, ANP, and OFP for peer-agent federation (Admin → Agent Interop).

SubAgent Delegation

Use SubAgent Delegation to isolate deep work in child threads while the parent keeps context lean.

Agent Handoff

Use Agent Handoff to transfer the root thread to a specialist agent when the task surface changes.

Agent Interop

Use Agent Interop with A2A, ACP, ANP, OFP, and other standards to communicate with external agents.

  • Outbound delegation - in-app agents send A2A JSON-RPC tasks to registered peer connections with allowlisted origins and bearer authentication
  • Inbound discovery - workspace publishes an Agent Card at /.well-known/agent-card.json plus per-agent cards
  • Inbound API - bearer-authenticated JSON-RPC at /api/a2a/json-rpc for inbound task execution
  • Admin → Agent Interop - connection CRUD, allowlisted HTTPS origins, audit metadata on tool calls

Workflow-aware AI

Agents can search community workflow templates by semantic similarity and, with user consent, read workflow configuration from the canvas. Execution data is queried separately - configuration and runtime data remain strictly separated.

Workflow orchestration

Visual pipeline model

A workflow is a directed graph of nodes connected by edges. Each node performs one step and passes typed dataset ports downstream.

Execution engine

  • Reactive, hash-driven execution - nodes run when inputs or configuration change
  • Concurrent scheduling - independent branches run in parallel
  • Server-side runs - large files, PDF parsing, and connector fetch on the workflow engine
  • Run-scoped file store - uploads, Arrow datasets, and exports persist across restarts within the workspace boundary
Validations

Apply schema, business-rule, and tolerance checks as deterministic steps; failures route to owners with full context attached.

Transformations

Reshape, normalise, and map payloads between source formats and target schemas with versioned transformation logic.

Compute

Run grouping, filtering, aggregations, enrichment, and other compute steps on governed datasets inside the workflow boundary.

Node families (representative)

Operations and low-code transforms

The Operations node exposes column-level transforms (strings, dates, decimals, conditionals, lookups). Users can combine visual operations or use natural-language compile (AI-assisted) to generate operation stacks from plain English.

Document processing

PDF and office documents uploaded to File Input are parsed server-side via Docling on the platform cluster (shared document-parse service for all workspaces). Optional AI structuring enriches results for review in the UI.

Subgraphs and templates

  • Workflows can be saved as templates and shared within teams or the community catalog
  • Subgraphs encapsulate reusable pipeline fragments

Observability

  • Per-node status, row counts, and audit trails on the canvas
  • HyperDX on the platform cluster - centralized OTLP ingestion, logs, traces, and operator dashboards for all workspaces (not duplicated per tenant cluster)
  • Gatus - in-cluster health dashboard for service uptime

Data ingress and egress

Fontana connects to 600+ upstream and downstream integrations through governed ingestion, enrichment, and export - without ad-hoc scripts. Connector images are mirror-pinned to GHCR and vulnerability-scanned as part of the SOC 2 supply-chain control set.

Upstream sources (ingress)

Downstream destinations (egress)

Ingress

Data Replication

Ingest data from major databases, SaaS, files, APIs, streams, and custodian feeds into governed workflow datasets.

File Input - supported formats

Default File Input nodes accept the extensions below (nodes can restrict to a subset via acceptedFileTypes in node configuration).

FIX parsing

FIX (Financial Information eXchange) log files load through the same File Input → workflow engine path as other structured imports.

  • Input: .fix uploads route to Fontana FIX Parser with MIME text/x-fix. Each FIX message becomes one workflow row.
  • Delimiters: SOH (ASCII 1), pipe (|), and escaped \x01 field separators are accepted in log files.
  • Field names: Tags map through a static FIX 4.4 dictionary (from QuickFIX FIX44.xml) to human-readable column keys such as MsgType and SenderCompID. Unknown tags keep the numeric tag string; duplicate tags become arrays.
  • Message boundaries: Extractor finds segments from 8=FIX. through the 10= checksum field.
  • Schema UI: FIX rows are treated as already structured (one row per message) - the importer does not suggest subgrid flattening for FIX sources.
  • Export: Save File Export supports .fix via rowsToFixMessages. Downloads use SOH delimiters with recomputed BodyLength (tag 9) and CheckSum (tag 10); the canvas preview uses pipe delimiters. Export requires mappable columns plus BeginString and MsgType on each row.
  • Runtime metadata: Engine parses set metadata.engineFile on the File Input output port (type, size, name) for server-side runs.

Related finance formats on File Input - SWIFT (.swift, .mt940) and OFX/QFX - use separate parsers in the same package; FIX round-trip export is the finance format currently supported on Save File Export.

Mechanism Description
File Input Upload or drag-and-drop - see supported formats above. Tabular, FIX, SWIFT/MT940, and OFX/QFX files parse to row datasets on the workflow engine; PDF and office documents route through Docling on the platform cluster.
REST Input External systems POST JSON to trigger a run and populate the input node output port.
Data Connector (Airbyte) Airbyte-catalog sources via PyAirbyte - Notion, Postgres, S3, and hundreds of connectors. No separate Airbyte platform server.

PyAirbyte execution path

  • Python/YAML connectors run in-process in the runner sidecar
  • Docker/Java connectors spawn isolated, short-lived Jobs; images are mirror-pinned, digest-pinned, and Trivy-scanned
  • No runtime package installs - connectors are baked into pinned images at build time

Egress

Data Export

Automatically upload transformed data to internal or downstream warehouses, lakes, BI, accounting, and domain platforms.

Mechanism Description
Save File Export CSV, TSV, PSV, XLS, JSON, XML, TXT, and FIX (.fix round-trip); server runs persist files under the run export directory (download from the node or the Files UI).
REST API node Call external HTTP APIs; map responses into the workflow dataset.
GraphQL API node Execute GraphQL queries against external endpoints.
Data Connector (Airbyte) Catalog destinations and sources through the same PyAirbyte path as ingress.

Secrets for connectors and APIs

  • Connector secrets use {{secret.NAME}} placeholders in node configuration
  • Values resolve from per-workspace HashiCorp Vault at execution time - SOC 2-aligned secret storage with no plaintext in workflow config
  • Distinct from BYOK LLM keys (different resolution path and admin surfaces)

Audit and data lineage

Fontana records atomic data lineage across workflow execution: every transformation, lookup, merge, filter, compute step, and AI operation that derives a cell can attach pointer-based provenance back to upstream rows and columns. Lineage is stored as structured metadata alongside port datasets - not as copied row snapshots - so auditors can traverse a recursive source chain from any output cell to its origins.

  • Per-operation lineage - workflow processors emit DataLineage entries with usage types such as transformation, lookup, merge, reference, computed, generated, and filtered; function operations can declare additional lineage from compute results
  • Per-port artifacts - server runs persist .lineage.json and .audit.json sidecars under each node port in the run directory; validation failures and operation summaries surface as audit items on retained rows
  • UI inspection - the Data Lineage panel on the canvas walks the provenance tree for the selected cell; workflow data views expose rows, errors, lineage, and edits for the active run
  • AI and admin audit - tool invocations write to ai_tool_calls with protocol and latency metadata; Flow admin audit logs cover privileged configuration changes; optional optional OTLP export to HyperDX on the platform cluster for operator dashboards
  • Write-only secrets - connector, MCP, and BYOK credentials are stored write-only in Vault; they are resolved only inside server actions at execution time and are never returned to the browser or embedded in workflow config
  • SOC 2-aligned evidence - run-scoped audit and lineage files persist on the workflow-engine File Store PVC across pod restarts; deploy snapshots and pinned image tags provide an immutable platform change trail separate from per-run workflow evidence

Secure storage

Each workspace (tenant) is a separate k3d Kubernetes cluster with its own Postgres, Convex backend, HashiCorp Vault, Zitadel identity plane, and persistent volumes. There is no shared database or secret store across tenants on a host - cluster-per-tenant is the primary segregation boundary for data, credentials, and compute. Tenants on the same physical box share one Linux kernel and Docker daemon (soft multi-tenancy); regulated workloads requiring hard isolation graduate to a dedicated host with the same mechanism and a smaller blast radius.

Encryption at rest

Production hosts use AWS EBS volumes with encrypted = true (root plus a dedicated data volume with prevent_destroy). All tenant PVCs - Postgres, Convex backend data at /convex/data, workflow-engine File Store, and Vault Raft - are provisioned through the cluster local-path StorageClass on that encrypted data volume. Fontana does not apply a separate application-layer field encryption; protection relies on infrastructure encryption plus tenant isolation.

Workflow engine file store

The workflow engine persists run-scoped files on a dedicated Kubernetes PVC per tenant cluster, not ephemeral container storage.

  • Claim: workflow-engine-data (20Gi, ReadWriteOnce) with helm.sh/resource-policy: keep so Helm upgrades do not reclaim the volume
  • Mount: /var/lib/fontana/uploads (FONTANA_UPLOADED_DATA_DIR) on the workflow-engine container
  • Contents: user uploads, per-run Arrow datasets, audit and lineage sidecars, export files, edit overlays, and hydration pointers used during server-side workflow execution
  • PyAirbyte sidecar: the airbyte-runner container in the same pod mounts the same PVC path so connector Arrow output paths resolve for the engine over localhost
  • Deploy strategy: Recreate rather than RollingUpdate. The deployment runs one replica on a ReadWriteOnce PVC, so only one pod may mount the volume at a time; a rolling update would leave the replacement pod waiting for the prior pod to release the claim.
  • Scope path: run directories are namespaced under each workspace's upload root so tenant file data never shares a filesystem path with another cluster
  • Medium: local-path StorageClass on the box's encrypted, prevent_destroy EBS data volume - same durable layer as Postgres and Convex PVCs

Security summary

Production deployments are designed for SOC 2 diligence with SOC 2-aligned controls across identity, segregation, secrets, audit, and supply chain. See Security and governance for certification status and the diligence pack. Contact Fontana for control narratives and evidence under NDA.

Topic Approach
Authentication Self-hosted Zitadel per workspace - OIDC, SSO federation, MFA (TOTP + WebAuthn)
Provisioning SCIM v2 inbound provisioning; role assignment via Zitadel project roles
Authorization Application RBAC in Convex (@fontana/rbac) with team scoping and permission registry
Service authentication Dedicated Zitadel OAuth clients for workflow-engine and platform-api; RFC 7662 token introspection on engine/API paths
Secrets Per-workspace in-cluster Vault (KV fontana); seeded in-cluster - no plaintext seed files; BYOK for AI; connector secrets via Vault placeholders at runtime
Encryption in transit TLS on all public endpoints (Caddy edge + ACME); in-cluster HTTP behind single-origin path routing
Encryption at rest EBS encrypted = true data volume backing all tenant PVCs - see Secure storage
Workspace isolation Cluster-per-tenant control plane, data, secrets, and identity - see Secure storage
Network Host security group: SSH + 80 + 443 only; k3s API bound to 127.0.0.1; NetworkPolicy default-deny with scoped egress per workload
Pod & workload hardening Per-workload ServiceAccounts (no default SA); Pod Security baseline; CPU/memory limits; Vault Agent sidecars for secret injection
Audit Atomic workflow lineage, per-port audit sidecars, AI tool-call records, admin audit logs - see Audit and data lineage; optional OTLP export to HyperDX on the platform cluster
Backup & recovery Automatic full-stack snapshot before each tenant upgrade; fontana rollback for reversible deploys. Application composite backup exports Convex configuration; Vault Raft snapshot restores secrets/BYOK (composite backup omits BYOK). Postgres, Convex, and workflow-engine PVCs are in scope for volume-level DR procedures.
Supply chain Immutable GHCR sha-<git> image tags, bundle SHA256SUMS verification, Trivy CRITICAL gate - no monorepo source on prod boxes
Change control Manual pinned releases on prod; Terraform for host provisioning; SSM read/debug only - durable changes via Terraform and fontana apply
Availability monitoring In-cluster Gatus probes platform and tenant services (flow, workflow-engine, convex-api, Vault, Postgres, Docling)