Upgrades
The production procedure for moving an install to a new engine version (check in CI, back up, migrate, apply the schema, roll out, verify), how a rolling upgrade behaves across replicas, and how to roll back.
An upgrade is a new binary or image against the same project and database. The project usually needs no change; the database needs the new version’s migrations. Upgrading the engine explains what ties a project to an engine (engine.version, the template contract and the migrations). This page is the production procedure.
Before the upgrade
Read the release notes for every version between yours and the target: what changed, the migrations each brings and any project change it needs.
Check the project against the new engine in CI, with the new image or binary and the production environment:
ENVIRONMENT=production engine-dmo check --no-db
engine-dmo check --no-db --contract-report check exits 3 when anything is invalid. --contract-report lists template usage the next template contract removes, each with its replacement.
Rehearse on staging: restore last night’s production backup into staging, run the new engine’s migrate up against it, and note how long each migration took (migrate up prints each with its time and class). A bounded migration on a large table is the one to plan around.
The upgrade
Back up the database and note the backup’s name. Migrations have no down step.
See what will run, with the new binary:
engine-dmo migrate status # exits 2 when migrations are pending Apply the migrations and the schema, with the new binary and the production environment:
engine-dmo migrate up
engine-dmo schema apply Roll out the new binary to every replica (restart the service, or deploy the new image tag).
Wait for readiness on each replica: engine-dmo health --readiness --wait 120s, or your orchestrator’s readiness probe on /health?readiness.
Verify: engine-dmo version reports the new version; engine-dmo doctor shows no errors; the admin’s diagnostics screen and your error reporter are quiet.
Warm: engine-dmo warm repairs the search index, requests the pages the page cache warms and reports each page’s status and time. Add --media to queue image derivative generation for every published page.
Rolling upgrades with several replicas
Run migrate up and schema apply once, as a separate step (a Kubernetes Job, a pre-deploy task, a one-off container), before the new replicas start. Do not rely on migrate.on_boot: apply with several replicas: the first replica would migrate inside its own start-up, under the orchestrator’s start timeout.
The migration step runs once, between the old replicas and the new ones:
sequenceDiagram participant O as Old replicas participant J as One-off step (new binary) participant DB as PostgreSQL participant N as New replicas O->>DB: keep serving J->>DB: migrate up, under the migration lock J->>DB: schema apply Note over O: readiness checks only the<br/>migrations they know N->>DB: start: every known migration applied? DB-->>N: yes N->>N: health --readiness passes Note over O,N: the load balancer moves traffic to the new replicas O->>O: stopped
What each process does during the rollout:
| Process | Behaviour |
|---|---|
A second migrate up started at the same time | Waits on the migration lock, then finds nothing to do |
| An old replica still running | Keeps serving. Its readiness checks only the migrations it knows, which are all applied |
| An old replica that restarts | Starts: migrations newer than the binary are not treated as pending or drift. Whether its code is correct against the new tables is release-specific; the release notes say when old and new must not run together |
| A new replica | Starts once every migration it knows is applied; refuses (exit 2) if migrate up has not run |
Rolling back
| Situation | Roll back by |
|---|---|
| The new version brought no migrations | Deploy the previous binary or image tag |
| Migrations ran, and the release notes say the previous version runs on them | Deploy the previous binary or image tag |
| Migrations ran, and the previous version cannot run on them | Stop the engine, restore the backup taken before the upgrade, deploy the previous version. Content written since the backup is lost |
Keep the previous image tag and the pre-upgrade backup until the new version has run in production for long enough to trust it.
Related
Data protection duties
The data protection work that falls to whoever runs an install, answering a subject access request, generating the impact assessment, and turning on and verifying the audit log's hash chain, with what the chain cannot detect.
Monitoring
How to watch a running Meridian install, covering the /health liveness and readiness endpoint, the Prometheus metrics listener and its series, OpenTelemetry traces and metrics, Sentry error reporting, the JSON logs, the job queue and scheduler commands, and the doctor support bundle.