Schema changes

The type files are the source of truth; the database follows them. You never write a migration for a content change. Instead the engine compares the files with what the database last applied and shows you a plan; applying it changes the database’s record of the model in one transaction. Stored content is never rewritten or deleted by a schema change: a value a change cannot carry is kept and flagged.

The loopCopy Anchor Link

engine-dmo check --fix     # write ids for new elements, then validate
engine-dmo schema plan     # what would change; exit code 2 while changes are pending
engine-dmo schema apply    # apply it in one transaction

check needs no database (--no-db skips the checks that do). schema plan and schema apply need a reachable database in DATABASE_URL. Every command that loads the configuration, check --no-db aside, also needs each ${VAR} in engine.yaml without a default to be set, and the engine does not read a .env file: export them in the shell. Once engine.yaml has signing_key: ${SIGNING_KEY} for production, set a development SIGNING_KEY of at least 32 characters locally too, or those commands fail with server.security.signing_key must be at least 32 characters.

The dev field verbs (add, set, rename, remove) run without a database or DATABASE_URL. Each checks the project after its edit and writes nothing when the result would be invalid (the change would leave the configuration invalid; nothing was written, exit code 3). That check is check’s, so what check does not catch, such as a relation to: naming a type that does not exist, is written. In development, engine-dmo dev up serves the site and, on every change to a file, writes missing ids, checks, plans and applies what is safe, then reloads; engine-dmo dev status shows what it last found.

Reading a planCopy Anchor Link

An example plan:

Plan for meridian <version>, project config at config

  + type        walk                             custom, 6 fields
  + field       walk.title                       text
  + field       article.heroImage                asset
  > field       listing.bookingLink              -> bookingUrl; handle only
  ~ field       event.price                      text -> money [no exact coercion for 3 of 41 values]  [DESTRUCTIVE: 3 documents hold a value]
  - field       listing.faxNumber                text; values are kept under _orphaned  [DESTRUCTIVE: 12 documents hold a value]
  ~ type        page                             presentation, validation or routing

Plan: 1 add type, 2 add field, ...

Resources include locale, site, site_locale, type, field, block, option, variant, singleton (a site’s singleton document apply will create), and the roles, grants and groups of config/access. engine-dmo schema plan --json prints the changes with op, resource, key, detail, destructive and affected, and the plan’s digest.

What apply does with a planCopy Anchor Link

flowchart TD
  start["engine-dmo schema apply"] --> valid{"configuration valid?"}
  valid -- no --> exit3["refused, exit 3"]
  valid -- yes --> diff{"any changes?"}
  diff -- no --> done0["nothing to do, exit 0"]
  diff -- yes --> outright{"handle reused under another id,<br/>a lockout, or NOT YET?"}
  outright -- yes --> refuse["refused: fix the files<br/>or pass --replace"]
  outright -- no --> destr{"destructive?"}
  destr -- no --> apply["applied in one transaction"]
  destr -- yes --> approved{"approval file digest matches,<br/>or --allow-destructive confirmed?"}
  approved -- yes --> apply
  approved -- no --> exit2["refused, exit 2"]

What is destructiveCopy Anchor Link

A change is destructive when documents hold values it affects. The plan counts them ([DESTRUCTIVE: 12 documents hold a value]); the same change with no values affected is not destructive.

Never destructive: adding a type, field, block, option or variant; a rename; labels, help, groups, validation rules, visible_when, ordering and templates (shown as presentation, validation or routing); turning index on or off (a reprojection job).

Approving a destructive planCopy Anchor Link

schema apply refuses a destructive plan unless it is approved, in one of two ways:

• Interactively: engine-dmo schema apply --allow-destructive prints the plan and asks you to confirm on the terminal. In a script with no terminal, add --yes.
• By file: write the plan’s digest (printed at the end of a destructive plan, and as digest in --json) to config/schema/.destructive-approved and commit it. Apply then accepts exactly that plan; any other plan is refused. The path is schema.destructive_approval in engine.yaml.

A refused apply prints the plan and refused: write the digest above to config/schema/.destructive-approved, or pass --allow-destructive, and exits with code 2.

What apply refuses outrightCopy Anchor Link

Apply recomputes the plan inside its transaction under a lock, so two applies at once cannot double up. Every apply, refused or not, is recorded in the audit log.

Renaming by idCopy Anchor Link

Every type, field, block, option, group and variant has an id: in the file. The database keys content by those ids, so a handle is free to change: rename a field by changing its name: and keeping its id:. The plan shows > field listing.bookingLink -> bookingUrl; handle only and no stored value moves.

Templates and API clients may still use the old handle. Keep it answering, with a warning, by adding it to aliases:

fields:
  - { name: bookingUrl, id: fld_7s2kq9m4xd, aliases: [bookingLink], type: link, kinds: [{ of: url }] }

engine-dmo dev field rename does all of this, and rewrites validation rules that name the field:

engine-dmo dev field rename listing bookingLink bookingUrl --dry-run   # print the diff
engine-dmo dev field rename listing bookingLink bookingUrl              # write it, keeping bookingLink as an alias
engine-dmo dev field rename listing bookingLink bookingUrl --no-alias   # or without an alias

When nothing uses the old handle any more, retire the alias:

engine-dmo schema alias list
engine-dmo schema alias retire listing.bookingLink

retire is refused while a template still uses the alias. A handle cannot be both an alias and a field: retire the alias before a new field takes the name. Renaming a select option’s value, a block’s key or a type’s key works the same way: keep the id:.

If you remove an element’s id: and run check --fix, it gets a new id and the plan treats it as a new element, not a rename. An id written on two elements fails check: id fld_... is written on two elements: field "endDate" here and field "startDate" at ....

Removing a fieldCopy Anchor Link

engine-dmo dev field remove listing faxNumber --dry-run

Removing a field never deletes data. Each document’s value stays in its revisions under _orphaned, keyed by the field’s id. In the admin, the editor’s advanced issues panel lists a document’s orphaned values and can recover one into an empty field of a compatible type, including a field re-added later with the same id:. dev field remove refuses while a validation rule or another element still names the field.

Removing a block keeps its items in the revisions too; switching a document’s variant sets aside the fields the new variant lacks in the same way.

Retyping a fieldCopy Anchor Link

Changing a field’s type: keeps its id, and the plan counts the values that have no exact coercion to the new type. Values are never converted in the database: each is read through the new type, and one that does not fit is kept as it was and reported to the editor, who can fix it.

Everything else is impossible: the value is kept and flagged. Restoring an old revision written under an earlier schema runs the same reconciliation and shows the editor a preview, field by field (carried, coerced, needs attention, orphaned, added).

ReprojectionCopy Anchor Link

When a change alters what is projected for queries (a field retyped, removed, indexed or unindexed, moved between chains, or an option removed), apply queues a reprojection job per type, which runs after the commit and resumes where it stopped. To run one by hand:

engine-dmo content reproject --type listing

Applying on startCopy Anchor Link

When the server starts, it compares the files with the database. schema.on_drift in engine.yaml decides what happens when they differ:

Run engine-dmo schema apply as an explicit deploy step after engine-dmo migrate up (Deploying).