Sources and importers

Two kinds of inbound integration need only configuration:

• A source is another service’s JSON that a page or an endpoint reads at request time: a weather forecast, a tide table, a booking system’s availability. The engine fetches it, caches it and protects the page from the service failing.
• An importer fetches a JSON feed on a schedule and turns each item into a document of one of your types, matched by the item’s id in the feed, so a second run updates what the first created. It never publishes.

Both fetch only the addresses written in configuration. A request can never make the engine fetch an address it supplies, and a redirect is followed only to the same scheme and host.

engine-cms behaves the same.

SourcesCopy Anchor Link

# config/sites/<site>/sources.yaml
sources:
  weather:
    url: https://api.example.org/forecast?lat=52.06&lon=-1.34
    headers:
      Authorization: "Bearer ${WEATHER_TOKEN}"
    ttl: 10m
    timeout: 3s
    max_bytes: 256KB
    on_error: stale_then_empty
    circuit_breaker: { failures: 5, window: 1m, cooldown: 30s }

Run engine-dmo check --fix to give it its stable id (src_...). Rename one with engine-dmo dev source rename <old> <new>, which also updates every endpoint step that names it.

The answer must be JSON with a 2xx status. Each engine process caches each source in memory for ttl, and fetches it once at a time: a burst of requests after the ttl lapses waits for the first fetch rather than each calling the service. The cache is not shared: each replica fetches for itself, and a restarted process starts empty, with no last good answer for the stale_ choices below to fall back on until its first successful fetch.

Empty is nothing (null). A failure fails an endpoint’s source step with HTTP 502 (“This information is not available just now. Try again in a moment.”); in a template, source() is simply empty.

In a templateCopy Anchor Link

{% set forecast = source('weather') %}
{% if forecast %}
  <p>{{ forecast.summary }}, {{ forecast.temperature }}°C</p>
{% endif %}

source('name') returns the parsed JSON, or nothing when the source is not declared or cannot answer, so always handle the empty case. See Functions.

In an endpointCopy Anchor Link

# config/sites/<site>/endpoints.yaml
endpoints:
  - key: forecast
    path: /api/forecast.json
    methods: [get]
    access: public
    steps:
      - source: { name: weather, as: forecast }
    respond:
      json: { from: forecast, fields: [summary, temperature] }
    cache: 5m

The source step takes name (a source in this site’s sources.yaml) and as (the name later steps and respond read it by; the source’s name when left out). respond: json: exposes only the fields it lists. See Forms and submissions for endpoints.

ImportersCopy Anchor Link

# config/sites/<site>/importers.yaml
importers:
  partner-events:
    url: https://feeds.example.org/events.json
    headers:
      Authorization: "Bearer ${EVENTS_FEED_TOKEN}"
    schedule: "0 4 * * *"
    type: notice
    locale: en-GB
    items: data.events
    external_id: id
    map:
      title: name
      summary: description.short
      startDate: dates.start
    status: in_review
    max_items: 500

Here notice is a type of the project’s with title, slug, summary (text) and startDate (date) fields. With a feed like this, each object under data.events becomes a notice document in en-GB, its title from name, its summary from description.short and its startDate from dates.start:

{ "data": { "events": [
  { "id": "ev-1042", "name": "Summer fair", "description": { "short": "Stalls, music and a dog show." }, "dates": { "start": "2026-07-18" } }
] } }

Run engine-dmo check --fix to give it its id (imp_...). Its import records are keyed by that id, so a renamed importer (engine-dmo dev importer rename <old> <new>) keeps matching the documents it made.

What a run doesCopy Anchor Link

For each item:

1. It reads the external id. An item without one is skipped and reported.
2. It reads each mapped field. When slug is not mapped, the slug is made from the mapped title (or item-<external id> when there is none).
3. If nothing mapped has changed since the last run, the item is unchanged.
4. If the item was imported before, the document’s working revision in the locale is updated: the import owns the fields it maps and nothing else, so an editor’s work on any other field survives the next run. A document that was trashed or purged since is created again.
5. Otherwise a document is created, owned by the site.
6. With status: in_review, the document is sent to review.

The values are validated as an editor’s save is; an item whose values fail is skipped with the reason. Each revision is recorded as written by the import (Imported by <key>). A run reports how many items were created, updated, unchanged and skipped, with up to 50 problems, in the job’s progress.

Running an importer nowCopy Anchor Link

Each importer is a scheduled task named importer.<site>.<key>:

engine-dmo schedule list --project .
engine-dmo schedule run importer.main.partner-events --now --project .

A staff endpoint can also queue a run with a job step, which is the one job kind an endpoint may queue:

steps:
  - job: { name: importer.run, with: { importer: partner-events } }

A burst of runs of one importer collapses into one.