Storage and media

Every uploaded file and every image derivative the engine generates lives in one blob store, chosen by storage.driver. Everything else (the media library’s records, alt text, rights, focal points) is in the database. A backup is therefore the database and the store together.

Choose a driverCopy Anchor Link

At boot the engine proves the store works before it serves: a filesystem root must be writable, and a bucket must accept a write, a read back and a delete within 30 seconds. A store that fails stops the boot with an error naming the store and the step that failed, not the first upload.

Filesystem storageCopy Anchor Link

storage:
  driver: filesystem
  filesystem:
    root: ./storage

The root must be writable by the engine’s user (uid 65532 in the container images). The engine creates what it needs below it:

storage/
  originals/ab/ab12...e9.jpg            # each upload, named by its SHA-256
  derivatives/<asset id>/<hash>/1200.webp

With server.replicas above 1, the filesystem driver is refused unless storage.filesystem.shared: true says every replica sees the same files: an upload that lands on one replica’s local disk is missing on the others. shared: true with one replica is a warning.

S3-compatible storageCopy Anchor Link

storage:
  driver: s3
  s3:
    endpoint: ${S3_ENDPOINT:-}
    region: ${S3_REGION:-eu-west-2}
    bucket: ${S3_BUCKET}
    access_key_id: ${S3_ACCESS_KEY_ID}
    secret_access_key: ${S3_SECRET_ACCESS_KEY}

The credentials need to put, get, delete and list objects in the bucket. The bucket does not need to be public: the engine reads from it and serves media itself. A plain http endpoint is a warning in production unless it is on a private network the engine alone reaches.

How media is servedCopy Anchor Link

The engine serves every media file from its own /media/ path, on every site’s host:

A derivative is generated on its first request and stored. While it cannot be generated, the request is answered with the original (which the browser scales), the failure is logged and counted, and the next request tries again. Byte ranges are answered from the store, so audio and video seek and a large PDF loads in parts. Originals are served so that they cannot run script on the site’s origin.

The hash is signed with media.signing_key, or a key derived from server.security.signing_key when that is unset, so nobody can ask the engine to generate a crop or size your templates never use. Changing either key changes every derivative address. Images covers transforms, widths and formats.

A CDN in frontCopy Anchor Link

Because every media address is immutable, any CDN can cache /media/ for a year. Two ways to put one in front:

Upload limitsCopy Anchor Link

A reverse proxy in front of the engine has its own body limit (nginx’s client_max_body_size is 1MB by default); raise it to at least media.max_upload_bytes. Caddy has no limit by default. HEIC uploads need the -vips image or a build linked against libvips.

Keeping derivatives in checkCopy Anchor Link

Derivatives accumulate as templates and transforms change. A scheduled task, media.prune_derivatives, runs daily and removes:

• every derivative of a deleted asset;
• derivatives not served or warmed for media.derivatives.keep_unused (default 90d);
• once unused for a day, derivatives of an old version of an asset, or of a transform definition nothing produces any more.

Run it by hand, and see the store’s size:

engine-dmo media stats                    # the library's size, derivatives by transform and format, alt text coverage
engine-dmo media prune --dry-run          # what would go, and the bytes it would free
engine-dmo media prune --older-than 30d

After a deploy that changes transforms, generate the derivatives the published pages use before visitors ask for them:

engine-dmo media warm --dry-run           # count what is missing
engine-dmo media warm                     # render every published page in process and generate what they reference
engine-dmo media warm --shard 0/4         # one quarter of the pages; run 1/4, 2/4 and 3/4 elsewhere in parallel

Moving between driversCopy Anchor Link

There is no built-in copy between drivers. To move from filesystem to s3, copy the whole storage root into the bucket with its paths unchanged (for example with aws s3 sync ./storage s3://<bucket>/ or rclone copy), switch storage.driver, and restart. Stop uploads (or the engine) while you copy, so nothing lands in the old store after the copy starts.

RelatedCopy Anchor Link

• Backups and restore: backing up the store with the database.
• Scaling: storage for several replicas.
• Media: the media library itself.