Skip to content
Back to

Backup and restore

Fontana separates three recovery questions that used to be answered by one mechanism, because they fail in different ways and on different timescales:

Question Path
“The new release is wrong.” Release rollback: pin the previous source.tag and run fontana apply
“We lost data, or a volume, or a cluster.” Nightly backups in your bucket, taken while everything keeps running
“Someone changed the configuration and we want the old one.” Admin configuration export and import in Flow

Keeping them separate means an upgrade never has to stop your workspace to be safe, and a backup window is not a maintenance window.

Backups run online

Section titled “Backups run online”

Every workspace runs a nightly backup job inside its own namespace. It exports each stateful component from the running workspace and uploads the set to an S3-compatible bucket you own. Nothing is stopped and no page is shown to your users.

What is protected How
Relational data Logical dumps of every Postgres database in the workspace: the backend, identity, and lineage
Secrets A logical export of the workspace’s Vault
Knowledge Graph A point-in-time snapshot of the graph database
Backend data An export of the backend’s documents, indexes, and stored files
Uploaded files and deployed functions The workspace’s uploaded files and deployed functions are part of every backup, copied from the backend’s storage volume right after the database dump so the two always agree
Workflow files and documents One versioned bundle per file-store collection, including history
Run manifest Written last. A run without it is incomplete and is never counted for retention

A run takes well under a minute on a typical workspace. Each run lands under its own timestamped prefix: <backupPrefix><workspace>/<timestamp>/.

Declaring the bucket and retention
Section titled “Declaring the bucket and retention”

Backups are configured in fontana.yaml and switched on by the presence of remoteStore. Without it, no backup job is created.

remoteStore:
  kind: s3
  endpoint: https://s3.example.com
  region: eu-central-1
  bucket: acme-fontana-backups
  backupPrefix: backups/ # optional


snapshots:
  schedule:
    daily: "02:00" # UTC
  retain:
    daily: 7 # nightly runs kept
    monthly: 12 # months whose first run is kept

The bucket credentials are environment variables in the shell that runs fontana apply, never values in the file. Apply fails rather than deploy a job that cannot upload.

Retention runs inside the job, not as bucket lifecycle rules, so the policy lives in fontana.yaml with the same vocabulary on every install and works on any S3-compatible endpoint. After each run the job keeps the newest retain.daily runs plus the first run of each of the newest retain.monthly months, and removes the rest.

Restore

Section titled “Restore”

A restore is one command, run by your platform team from the workstation that runs fontana apply:

Terminal window
fontana restore <workspace> --run <timestamp> --yes

Leave out --run to take the newest complete backup. The command then does the whole recovery inside the cluster:

  1. Every service in the workspace is paused so nothing writes while the data is replaced.
  2. The backup run is downloaded and every artifact is checked against the manifest’s checksums before any data store is touched. A run whose manifest is missing never finished and is refused.
  3. The databases are replayed from their dumps, the workspace’s uploaded files and deployed functions are placed back on the backend’s storage volume, the secret store’s key-value contents are written back, and the graph snapshot and file-store collections are placed on their volumes.
  4. The services resume at their previous scale and the same health check that follows fontana apply must pass.

The --yes flag is required because a restore replaces the workspace’s data with the chosen point in time. Signed-in users are logged out by the restore and sign in again afterwards.

Because a restore puts data back into the same layout that fontana apply creates, recovering into a rebuilt cluster is the same procedure: provision, apply, restore. The workspace’s secrets are part of the backup, so restored services and restored databases agree.

Release rollback

Section titled “Release rollback”

Releases are immutable and content-addressed, and every installed release stays on disk beside the current one. To go back, point source.tag at the earlier release in fontana.yaml and run fontana apply. Workloads roll back the same way they rolled forward: a replacement starts, passes its readiness check, and only then does the current pod retire.

Schema changes are written to accept both the previous and the current shape for one release, and migrations run idempotently during apply. That is what makes a rollback safe to perform without restoring data.

A failed rollout does not need a manual rollback at all: if the new pods never become ready, the old pods keep serving. See Fontana CLI.

Application configuration export and import (Admin)

Section titled “Application configuration export and import (Admin)”

In Flow, Admin → Deployment → Export/Import exports and imports a composite configuration file for your workspace. Use it to clone settings across environments, recover after a configuration mistake, or document approved platform state.

Export includes (non-secret application configuration):

  • Roles, teams, and users (users merge by email on import)
  • Workflows, workflow documents, and schedules
  • AI agents, skills, skill assignments, and approved model lists
  • Preference definitions and overrides
  • Evals, saved subgraphs, and sidebar collection order
  • Knowledge Graph files and settings (namespace spend resets on import)
  • MCP and external agent interop connections
  • Lineage destination and endpoint
  • Slack and Teams chat-channel installs (imported disabled so you can paste tokens before enabling)
  • Non-secret platform configuration keys

Imported Knowledge Graph files need processing on the destination before the graph fills in.

Export does not include:

  • BYOK and other Vault secrets: LLM keys, identity SMTP credentials, chat and lineage tokens, and other sensitive values stay in Vault. They are in the nightly backup, not in the composite export.
  • Workflow run datasets: tabular run data and exports live in workspace storage (see Storage and processing) and are covered by the nightly backup.
  • Inbound email routes: mailbox addresses stay with the destination install.
  • Runtime history: chat transcripts, embeddings, and caches.

Import shows a preview with per-section choices: skip duplicates, overwrite, or import as duplicate. You control which sections apply before committing. If a section batch returns errors, import fails closed: later sections do not continue, and you see a failure message rather than a partial apply that proceeds quietly.

Requires platform:write permission, typically administrators.

When to use which path

Section titled “When to use which path”
Scenario Recommended path
Undo a bad release Pin the previous source.tag, run fontana apply
A release that never became healthy Nothing to do: the rollout stalls and the old pods keep serving
Recover workflow run history, uploads, or a database Restore from the nightly backup
Recover an entire cluster Provision, fontana apply, restore from the nightly backup
Migrate workspace configuration to another workspace Admin Export/Import composite file
Recover secrets and BYOK The secret-store export in the nightly backup, plus your escrowed unseal material
Change control before a planned change Manual export, and an on-demand run of the backup job if you want a fresh point in time

Upgrades and change control

Section titled “Upgrades and change control”

Production deploys use immutable release tags, so every upgrade is reproducible and auditable, and the pinned tag in fontana.yaml is the record of what ran. Because upgrades do not stop the workspace and backups do not stop it either, your change window is about approval and verification rather than downtime: apply the release, watch fontana status, and validate in the workspace.

Section titled “Related documentation”