Monitoring

The engine reports on itself five ways: a health endpoint for load balancers, Prometheus metrics on a listener of their own, optional OpenTelemetry export and error reporting, structured logs on standard error, and engine-dmo doctor for a one-file snapshot.

HealthCopy Anchor Link

Every site’s host answers GET /health:

Otherwise /health?readiness answers 503 with status set to starting (still booting or warming the page cache), stopping (shutting down) or degraded (a check failed), and the failing check’s error under checks. Responses are Cache-Control: no-store.

Use liveness for “restart it if it stops answering” and readiness for “send it traffic”. Readiness turns false as soon as a shutdown begins, so a load balancer drains a replica before it stops; the engine then finishes in-flight requests for up to server.shutdown_grace (30 seconds by default).

From the command line, against the server this project’s configuration describes:

engine-dmo health                     # liveness; prints the body
engine-dmo health --readiness         # exit 0 only when ready
engine-dmo health --readiness --wait 60s   # keep trying for 60 seconds, for a deploy step

engine-dmo health requests http://<server.listen>/health, so it runs on the same machine or in the same container as the server, with the same environment. It exits 4 when the server does not answer 200.

MetricsCopy Anchor Link

The engine serves Prometheus text metrics on a listener of its own, never on the public sites:

telemetry:
  metrics:
    enabled: true          # default
    bind: 127.0.0.1:9090   # default: this machine only
    path: /_metrics        # default

curl -s http://127.0.0.1:9090/_metrics

To let a Prometheus on another host or container scrape it, bind to an interface that network reaches (0.0.0.0:9090 inside a container network) and keep the port off the public internet. bind may not equal server.listen. Only engine-dmo serve opens the listener; one-off commands do not.

SeriesCopy Anchor Link

Series that read the database are collected at scrape time with a time limit, so a database that does not answer shows as meridian_db_up 0 rather than a scrape that hangs.

What to alert onCopy Anchor Link

OpenTelemetryCopy Anchor Link

Traces and metrics can also be pushed over OTLP/HTTP (JSON encoding) to a collector:

telemetry:
  otlp:
    endpoint: https://otel-collector.internal:4318   # /v1/traces and /v1/metrics are appended
    headers:
      Authorization: ${OTEL_AUTH_HEADER}
    timeout: 10s
  traces:
    enabled: true
    sample_ratio: 0.1
  metrics:
    export: true
    interval: 1m
  resource:
    service_name: meridian
    attributes:
      deployment.environment: production

Every header value is a secret (${VAR}). Traces sample 10% of new traces by default; an inbound traceparent decides only when respect_parent_sampling: true, so a visitor cannot make every request expensive. Each database statement is a span when tracing is on. The full key list is in engine.yaml: telemetry.

Error reportingCopy Anchor Link

Errors and panics can be reported to Sentry (or any service that accepts the Sentry protocol):

telemetry:
  errors:
    driver: sentry
    dsn: ${SENTRY_DSN}
    environment: production   # defaults to engine.environment
    sample_rate: 1.0          # errors only; a panic is always sent

Check the reporter and the OTLP endpoint from the deployment:

engine-dmo telemetry test --note "deploy 2026-11-02"

The engine also keeps its own grouped error log in the database, which engine-dmo doctor --include-logs 20 prints and the admin’s diagnostics screen shows. A configured secret quoted in an error is scrubbed before it is stored.

LogsCopy Anchor Link

The engine writes one structured line per event to standard error: JSON by default, or logfmt-style text with logging.format: text.

{"time":"2026-11-02T03:15:01.204Z","level":"INFO","msg":"request","request_id":"01JB...","event":"http.request","method":"GET","path":"/whats-on/","status":200,"bytes":48211,"duration_ms":12.4}

Each request is one request line with method, path, status, bytes and duration_ms, plus request_id and, when tracing, trace_id. A line written by a job or a provider call carries the request_id of the request that caused it. The request id is taken from an inbound X-Request-ID header when it comes from a trusted proxy, otherwise minted, and is sent on outbound calls. Single-use tokens in paths (enrolment, reset, magic and invitation links) are logged as {token}.

Boot logs the version, variant, bundle, modules, environment, the extensions loaded, the database server version, and each migration or schema change applied.

Jobs and scheduled tasksCopy Anchor Link

Background work runs in every engine process: jobs from a queue in the database, and scheduled tasks on whichever replica holds the scheduler lease.

engine-dmo jobs list                    # jobs not yet complete, and dead ones, newest first
engine-dmo jobs list --failed           # dead jobs and jobs waiting to retry after an error
engine-dmo jobs retry --id <job id>     # run a dead job again with fresh attempts, once the cause is fixed
engine-dmo jobs retry --all-failed --queue <queue>

engine-dmo schedule list                # every scheduled task, its cadence, last success and the leader
engine-dmo schedule run analytics.rollup --now   # run one task once, in this process; audited

schedule list reads the last successes from the running engine’s metrics listener, so run it where telemetry.metrics.bind is reachable.

With audit.hash_chain on, the daily audit.anchor task logs an audit chain head line per realm, with realm, seq and hash. Keep these lines: they are how a rebuilt chain is detected.

The doctor bundleCopy Anchor Link

engine-dmo doctor writes one pasteable text file describing the install: build, localisation data, configuration (non-secret), plugins, database (version, encoding, collation, ICU, pool, largest tables), storage, media pipeline, templates, mail, search (and whether the index matches what is published), jobs, permissions, timings, and recorded errors. It works without a database too, reporting the connection failure.

engine-dmo doctor --out doctor.txt
engine-dmo doctor --include-logs 20 --redact-level paranoid --out doctor.txt
engine-dmo doctor --mail-to you@example.org      # also send a live test message
engine-dmo doctor --telemetry-test               # also test the error reporter and OTLP endpoint

RelatedCopy Anchor Link

• engine.yaml: telemetry and logging
• Scaling: the scheduler lease across replicas.
• Error codes: every MRD-nnnn code a log line or refusal carries.