Inbound Webhooks (receiving events)

The mirror image of the Webhooks section above: instead of the Data Lake POSTing your row changes out, a third-party system POSTs events in and they land in your tables. Create an Import connection with the Webhook provider (no key to paste), add one or more endpoints to it, and give each endpoint’s URL to the sending system — WhatsApp Cloud API, Stripe, GitHub, Shopify, another Data Lake, or your own service. Target tables are named hook_<entity>.

A webhook connection has no Run, Resync or Verify — on purpose. Rows arrive when the sender posts, so there is nothing to start; the connection page says so instead of showing dead buttons. And a push source can’t be asked for everything it holds (nobody can ask WhatsApp for every message it would ever have sent you), so the “what the source has” side of a mirror or integrity diff can’t be built at all — a diff could only conclude the source is empty, which for a mirror would mean deleting your live rows. Webhook targets are therefore append-only. The health question a receiver can answer is “is anything arriving?”, which is what Test (does at least one enabled endpoint exist?), the per-endpoint received count / last-received time, and the Inbox tab answer.

Creating an endpoint. Pick a name and a verification profile, and the endpoint is created with its URL and signing secret shown once — copy them into the sending system straight away, because the platform stores only a hash of the URL and never the secret in readable form (support can confirm a URL you paste back, but nobody can recover it, and a stolen database is not a stolen set of webhook URLs). Two URL shapes work: the canonical https://hooks.datalake.commercient.com/hooks/<token> — the one to hand a vendor, since it survives a tenant rename — and the tenant form https://<slug>.datalake.commercient.com/api/ddl/hooks/<token>. Each endpoint has an Enabled switch, which is the instant revocation control, and a body-size cap (1 MiB default, 4 MiB maximum) — an oversized delivery is rejected with a 413 naming the cap.

Verification profiles. Every profile verifies the signature over the exact bytes received, and a delivery that fails verification is rejected with 401 and stores nothing (so somebody who learns your URL still can’t fill your table); the endpoint’s fail streak and last-fail time appear in the list, which is how you spot “the sender rotated its key at 14:02”.

• Signed (datalake) — our own outbound scheme (X-DataLake-Signature: t=…,v1=…, 5-minute window). Point another Data Lake’s outbound subscription at it with no glue.
• Bearer (bearer) — the sender sets one header (X-Webhook-Secret or Authorization: Bearer …). The recommended floor for a simple sender.
• Meta / WhatsApp Cloud API (meta) — X-Hub-Signature-256 keyed by the Meta app secret, plus Meta’s subscription handshake: Meta GETs the URL first with hub.mode=subscribe, and the endpoint echoes hub.challenge back as plain text only if the hub.verify_token matches. A wrong token gets a 403 without the challenge, so nobody can subscribe an app they don’t own to your URL.
• Stripe (stripe) — Stripe-Signature (t=…,v1=…), with a 5-minute timestamp tolerance and any of several v1 values accepted, so a Stripe secret rollover isn’t an outage.
• GitHub (github) — X-Hub-Signature-256, with GitHub’s event name and delivery id kept alongside each entry.
• Shopify (shopify) — X-Shopify-Hmac-Sha256 (Base64).
• None (none) — no verification at all: anyone who learns the URL can append rows. Because of that it is an explicit opt-in — the create dialog makes you type I UNDERSTAND verbatim, and the choice is written to the audit log. Use bearer or datalake instead unless the sender is on a trusted network.

Vendor secrets. The secret shown at creation only verifies a sender you configure; to verify a real vendor you paste its material into the endpoint’s secret slots — Meta’s app secret and hub.verify_token, Stripe’s whsec_…, a GitHub webhook secret, a Shopify shared secret (which slots exist depends on the profile). The form is write-only: values are never shown again, each slot just reads set or not set, a slot you leave alone is preserved, and Clear deletes one. Clearing the last usable secret is refused (it would turn every delivery into a silent 401 — disable the endpoint instead), and if somebody else changed the stored secrets while your form was open the save is rejected rather than overwriting them. Replacing an app secret or signing secret is a rotation: the old value keeps verifying for 24 hours — the vendor signs with whatever key its console had at send time — and the panel shows “previous key still accepted until …” with a clear-now option. The Meta hub.verify_token has no grace period, because its handshake is interactive and a stale token would only hide a mistake.

Rotating the URL. Rotate mints a new URL (shown once); the old URL keeps working for 7 days so the vendor’s console can be updated by a human without dropping deliveries. Rotating the signing secret alone leaves the URL untouched.

Mapping (event JSON → columns). Each endpoint carries a small JSON mapping document that says only which target table and which fields: a root path selecting where the rows are in the envelope (with [*] fanning one envelope out into several rows), a columns map of column name → JSON path (plus optional SQL type, required, default), an optional eventIdPath naming the sender’s own event id, and an optional filter to ignore envelopes you don’t want. There is deliberately no merge key and no insert/update/delete flag to configure: webhook tables are event logs keyed on event_id, which the platform supplies (falling back to a hash of the body when the sender has no id), and each row also gets _received_utc and _inbox_id — those three names are platform-owned and can’t be mapped over. The editor loads the document that is actually running, validates paths and types as you save rather than at 3 a.m., and Preview dry-runs it against a sample payload to show the columns and first rows it would produce. Because “what is true now” is a view over the event log (e.g. latest status per message) rather than an overwritten column, correcting a derivation never touches the underlying facts.

Inbox tab. Every accepted delivery is stored raw before anything is parsed, then a background worker maps it and merges the rows through the normal ingest pipeline (so the target is an ordinary table afterwards: Data API, RLS, change tracking, Time Travel, events all apply). The Inbox tab shows counts by status, a status filter and a paged newest-first list; opening an entry shows the pretty-printed payload and headers. Each entry ends on its own as drained (merged), failed (the reason is on the row — this is the dead letter, and it stays in place), skipped (a mapping filter excluded it — visible, not silently dropped), or is still new/claimed. One bad payload never fails its batch-mates. Transient problems retry themselves on a widening backoff (1 minute, then 5, 30 and 120, up to five attempts); a payload that simply can’t be mapped fails immediately, and after fixing the mapping you can Retry one entry or Retry all failed for the connection. A duplicate delivery is a no-op: identical bytes to the same endpoint are recognised and answered “already received”, so a vendor’s redeliveries (Meta, Stripe and Shopify all redeliver by design) never double-insert. There is deliberately no delete or purge — a failed envelope is the only copy of what the vendor sent; drained entries age out automatically (30 days by default).

Worked example — WhatsApp Cloud API. Create a Webhook connection, add an endpoint on the Meta / WhatsApp Cloud API profile, paste Meta’s app secret and your chosen verify token, and map root = $.entry[*].changes[*].value.messages[*], eventIdPath = $.id, with columns from $.from, $.text.body, $.timestamp and $.type. Save the endpoint’s canonical URL as the callback in Meta’s console — Meta GETs it, the endpoint echoes the challenge, and the subscription saves. From then on one envelope carrying two messages produces two rows in hook_wa_messages, each keyed on its own message id; a body tampered with after signing is rejected with 401; and an exact redelivery is answered as a duplicate with no second row. Delivery statuses (sent/delivered/read) arrive as their own envelopes about the same message — map them to a second target and they append as separate facts, with the full transition history (and delivery latencies) preserved.