Receive data from other systems

Most write paths into a Homespun app authenticate as the agent or a signed-in human. An inbound catch-hook is the exception: it gives an external system a secret URL to POST JSON to, and that JSON is written into one of your app’s collections with no agent online. Use it for Stripe events, a Zapier or Make push, a Home Assistant automation, or an email router that turns a message into a webhook.

Declare a hook in the manifest

Hooks are declared in x-homespun-manifest.ingest, validated at deploy exactly like webhooks, and materialized into a secret URL. There is no dashboard-created or agent-created hook: you add one by editing the manifest and redeploying.

"ingest": [
  {
    "name": "stripe-payments",
    "collection": "payments",
    "mode": "upsert",
    "upsertOn": "external_id",
    "map": {
      "external_id": "id",
      "amount": "data.object.amount",
      "customer": "data.object.customer"
    },
    "dedupeKey": "id",
    "wake": true
  }
]

Field by field:

• name (required): unique within the app, [a-z0-9][a-z0-9_-]{0,63}. This names the hook and survives redeploys: the same name keeps the same URL, so a sender you configured weeks ago keeps working.
• collection (required): the declared collection the incoming row is written into.
• mode: append (default) writes a new row per delivery; upsert merges onto an existing row.
• upsertOn: required when mode is upsert, and must name one of the collection’s unique fields. A redelivery with the same key updates the row in place.
• map (optional): target row field to a dot-path into the JSON body. Absent means the raw default row { hook, payload, receivedAt }, with the whole body under payload. A path that resolves to nothing omits that field, it never writes a null.
• dedupeKey (optional): where the value that dedupes a redelivery comes from, either a dot-path into the JSON body (the default form, e.g. "id") or header:<name>, a reference to a request header (e.g. "header:x-github-delivery"), for a sender whose replay id lives in a header rather than the body: GitHub puts its delivery id in X-GitHub-Delivery and its webhook bodies carry no equivalent stable id of their own. <name> is a lowercase HTTP header name; the lookup is case-insensitive regardless of how the header arrives. A second delivery carrying a value already seen is acked 200 and recorded as dropped_duplicate, not written again. Set this for any sender that retries (Stripe redelivers). This only ever guards an ACCEPTED delivery: a failed delivery releases its dedupe value, so retrying the SAME logical write under the SAME dedupeKey value after a failure (a fixed schema violation, or a stale if_match, see below) is a real write attempt, never a silently dropped duplicate. With no dedupeKey at all, every delivery (including an exact redelivery) writes a new row: the dedupe value is null, and null values never collide. A header:<name> dedupeKey never resolves during homespun ingest backfill, since a backfill replays stored bodies with no live request headers to read; a body-path dedupeKey still works there.
• handshake: set to echo to answer a Slack or Microsoft Graph URL-verification challenge.
• verify (optional): opt into body-signature verification. { "scheme": "github" } requires a valid GitHub HMAC-SHA256 signature over the raw body (see below). Strictly opt-in: without it the URL secret is the only check.
• wake: true auto-wakes a dormant app on delivery (default false).

Avoid lost updates with if_match

A backend that reads a row, computes for a while, and writes the result back through a mode: "upsert" hook has no way to tell “nobody touched this row since I read it” unless the write can carry the version it read. Send a top-level if_match in the POST body, the exact field name and type the direct PATCH /v1/apps/:id/collections/:name/:key route’s body already uses:

{ "id": "cust_42", "status": "reviewed", "if_match": 3 }

The delivery is checked against the row’s current version (the same counter a PATCH or DELETE’s if_match checks) before it writes:

• Match: the write lands, and the row’s version increments, exactly like a matching PATCH.
• Mismatch: the write is refused with a retryable 409 conflict, not acked 200, so a compute-then-write-back backend learns synchronously to re-read the row and retry rather than believing a stale write landed. This is the ONE other sender-visible failure past the guard checks, alongside the 503 a full app returns (see “What a sender sees” below).
• A malformed if_match (not a non-negative integer) is rejected as invalid_request and journaled failed, the same as any other malformed payload; it is never silently ignored, which would defeat the whole point of asking for it.

if_match is checked only for a mode: "upsert" hook, and only once the delivery’s natural key actually matches a live row. A mode: "append" hook always writes a brand-new row, so there is never an existing version for it to be stale against; an if_match key in an append body is inert. The very first delivery for a natural key (nothing to match yet) also ignores it, for the same reason: a fresh row has no prior version to compare against.

Everywhere if_match is not sent, the hook behaves exactly as before: last-write-wins. This is the default for every hook that does not use the field, and for every append hook regardless. Omitting it is not a regression; it is today’s behaviour, unchanged.

A corrective retry after a 409 should reuse the SAME dedupeKey value, if the rule declares one. It is the same logical write (re-read, recompute, resend), so reusing its identifier is the natural choice, and it is safe: a failed delivery (a version conflict included) releases its dedupe slot, so the retry is a real write attempt against the row’s current version, not a dropped_duplicate that never reaches the write. Only an accepted delivery keeps its slot.

Nullable fields

A mapped field arrives one of two ways, and the collection schema has to allow both:

• The source path is absent from the body: the field is omitted from the row (it is never written as null).
• The source path is present with an explicit null: the field is written as null.

So a sender that sends null for a field, rather than leaving it out, needs that field typed as nullable in your collection schema, or the delivery fails validation.

GitHub is the classic case. A workflow_job webhook sends "conclusion": null, "completed_at": null, and "runner_name": null on the queued and in_progress events, with the real values arriving only on completed. A collection that types those as a plain "string" accepts every completed delivery and rejects every queued / in_progress one with row_schema_violation, even though the payloads look identical in shape. Type any field a sender can send as null with a nullable JSON Schema type:

{ "conclusion": { "type": ["string", "null"] } }

When a delivery does fail this way, the failed delivery’s detail names the field and the received value, for example /conclusion must be string (received: null), so the cause is visible in the delivery log without diffing a working payload against a failing one. Fix the schema, redeploy, and replay the failed deliveries (see below).

Verify GitHub signatures

By default the URL secret is the whole authentication. For a GitHub webhook you can require a signature over the body as well, so a leaked URL alone cannot post forged deliveries. Add verify to the rule and provision a signing secret:

"ingest": [
  {
    "name": "gh-issues",
    "collection": "issues",
    "verify": { "scheme": "github" }
  }
]

1. Deploy the manifest. The hook is fail-closed until you provision its signing secret: every delivery gets 401 until you complete step 2.

2. Mint the signing secret and copy the value it shows once:

   homespun ingest signing-secret set --app <idOrSlug> --name gh-issues

3. In the GitHub webhook settings, paste that value into Secret, and set Content type to application/json (a application/x-www-form-urlencoded webhook gets 415, since the endpoint accepts JSON only).

The relay reads GitHub’s X-Hub-Signature-256 header, recomputes the HMAC over the exact bytes GitHub sent, and compares in constant time. A missing, malformed, or wrong signature (and a hook whose signing secret is not set) all return one uniform 401, and the rejected request is not recorded in the delivery log. Rotate the secret with homespun ingest signing-secret set again: the previous value keeps verifying for a grace window so deliveries do not drop while you update GitHub.

The URL

After deploy, read back the hook’s full secret URL:

homespun ingest list --app <idOrSlug>

Each hook returns a url shaped https://<relay>/v1/ingest/<hookId>/<secret>. The secret in the URL is the whole authentication: no header, no cookie. Hand that URL to the owner to paste into the external system, and treat it like a password. If it leaks, rotate it (the old URL stops working immediately, no redeploy needed):

homespun ingest rotate --app <idOrSlug> --name stripe-payments

An owner can also see the URL, a ready-made curl test, the rotate button, and the delivery log in the app’s dashboard under Inbound hooks.

What a sender sees

The endpoint is fire-and-forget. Past a handful of guard checks it always acks 200 {"ok":true} and records the outcome in the delivery log, so a sender never retries into a mapping bug forever. The failures a sender can see are 429 (rate or hourly cap), 404 (wrong URL, uniform so it is not an existence oracle), 415 (not application/json), 413 (body over the size cap), 503 (the app is at its row/storage quota, see the delivery journal for the retry story), and 409 (an if_match version mismatch, see “Avoid lost updates with if_match” above: re-read the row and retry).

Once accepted, the outcome is one of:

• accepted: the row was written (and flows through feed replay, live delivery to a connected agent, notify rules, and outbound webhooks like any other write).
• dropped_duplicate: a redelivery of a dedupeKey value an ACCEPTED delivery already used; no row written. A failed delivery’s value does not count as seen (it released its slot), so a corrective retry under the same value is never mistaken for this.
• failed: the write was rejected. For most reasons (a schema violation, an append-only conflict, a malformed if_match) the sender still got its 200; the failure is yours to see in the log. Two reasons are NOT acked 200, because a silent success would be actively wrong: the app being at its row/storage quota (503, see above) and an if_match version mismatch (409, see “Avoid lost updates with if_match” above). Both are still journaled failed here, so the log reads the same either way.

Inspect and replay

List recent deliveries, filter by hook or status:

# via the HTTP API (agent-or-owner auth)
GET /v1/apps/<id>/ingest/deliveries?hook=stripe-payments&status=failed&limit=25

Each delivery keeps a capped snapshot of the received payload. That powers the debugging loop: when a delivery failed because your map was wrong, fix the manifest, redeploy, and replay the stored payload through the corrected rule:

POST /v1/apps/<id>/ingest/deliveries/<deliveryId>/replay

Replay respects the current manifest: a hook renamed or removed since the original delivery gives a clean error rather than writing into the wrong place. A replay bypasses dedupe (you asked for it explicitly), so an append hook can produce a duplicate row on replay, while an upsert hook just merges onto the same key again.

The delivery journal is pruned automatically (by age and a per-app cap), so it is a rolling window for debugging, not a permanent store: the collection rows are the durable record.