Webhook Error Handling

Webhook delivery is at-least-once. A well-behaved endpoint will occasionally receive the same event twice, and must be idempotent. This page is the contract your handler should be written against.

The retry schedule

Four attempts maximum, at fixed offsets measured from the first attempt — not from the previous one.

There is no exponential backoff, no jitter and no extended retry window. Two minutes is the whole budget.

Scheduling guarantees

Delayed attempts are persisted jobs, not in-memory timers, so a process restart between attempt 2 and attempt 3 does not lose attempt 3. Two safeguards:

• Never early, never in the past. A scheduled time that has already passed when the job is created — for example after a restart — is clamped to one second from now rather than firing retroactively.
• A reconciliation sweep runs every 120 seconds and re-drives any pending delivery whose scheduled time is more than 60 seconds in the past, in deployment-tuned batches (600 per pass in the standard production configuration). The worst case for a delivery caught by an infrastructure hiccup is a late attempt, never a lost one.

When an attempt succeeds, the remaining scheduled attempts are cancelled immediately.

Timeouts

The 30 seconds covers connect, send, wait and read. An endpoint that takes longer records a timeout and the attempt is retried.

Redirects are not followed. The HTTP client is deliberately configured to ignore any proxy environment configuration, so deliveries always go directly to the host you subscribed.

What counts as success

HTTP 200 through 299 inclusive. Nothing else.

Your response body is not inspected. Returning 200 with {"error": "could not process"} is recorded as a success and the event will not be retried. If you cannot process an event, return a 5xx.

When retries are exhausted

After the fourth failed attempt the delivery is finalised as failed and carries a machine-readable reason:

exhausted_4_attempts:timeout
exhausted_4_attempts:http_503
exhausted_4_attempts:connect_error
exhausted_4_attempts:unknown

The suffix is the classification of the last attempt: http_<status> when we got an HTTP response, otherwise one of these error classes.

Deliveries that are never attempted

Some events are recorded as suppressed rather than attempted. Suppression records are kept for the consolidated event and call_corrected only, so that a missing terminal webhook always has an explanation.

Every delivery record — attempted or suppressed — is retained for 30 days, then removed automatically.

Duplicate deliveries

The delivery scheduler runs on every application process. Each attempt takes a distributed lock before firing, so exactly one process sends it. If a lock is lost to an expiry or a network partition, the cost is one duplicate POST — never a lost delivery.

That trade is deliberate: losing an event is unrecoverable, whereas a duplicate is free if you deduplicate. Which brings us to the next section.

Idempotency

Every event carries a stable identifier derived deterministically from the call and the event type:

event_id = UUIDv5(namespace, "{call_id}:{event_type}")

Which means:

• The same event always has the same event_id. All four delivery attempts carry byte-identical bodies. If the same event is ever emitted twice, its event_id is unchanged.
• (call_id, event_type) is the natural idempotency key, and event_id is simply that pair pre-hashed. Deduplicate on either — they are equivalent. Use event_id; it is one column instead of two and it is also available as the X-Graine-Event-Id header, so you can dedupe before parsing the body.
• call_id alone is not a key. One call can legitimately produce all_processing_completed and later call_corrected. Keyed on call_id alone you would discard the correction.

A worked handler

Insert the event_id into a unique-keyed table first. If the insert conflicts, you have already seen this event — acknowledge and stop.

// In YOUR database — example shown in PostgreSQL. Any store with a unique
// constraint works; see the note above.
//   CREATE TABLE webhook_events (
//     event_id   uuid PRIMARY KEY,
//     call_id    text        NOT NULL,
//     event_type text        NOT NULL,
//     received_at timestamptz NOT NULL DEFAULT now()
//   );
 
async function handleEvent(pool, event) {
  // Claim the event. ON CONFLICT DO NOTHING makes this a no-op the second time,
  // so a duplicate delivery costs one cheap insert and nothing else.
  const claim = await pool.query(
    `INSERT INTO webhook_events (event_id, call_id, event_type)
     VALUES ($1, $2, $3)
     ON CONFLICT (event_id) DO NOTHING
     RETURNING event_id`,
    [event.event_id, event.call_id, event.event_type],
  );
 
  if (claim.rowCount === 0) {
    return; // Already processed. Acknowledge with a 200.
  }
 
  // First time. `call_corrected` supersedes an earlier verdict for the same
  // call, so upsert on call_id rather than inserting a second row.
  await pool.query(
    `INSERT INTO calls (call_id, status, sub_status, cost, transcript, updated_at)
     VALUES ($1, $2, $3, $4, $5, now())
     ON CONFLICT (call_id) DO UPDATE SET
       status     = EXCLUDED.status,
       sub_status = EXCLUDED.sub_status,
       cost       = EXCLUDED.cost,
       transcript = COALESCE(EXCLUDED.transcript, calls.transcript),
       updated_at = now()`,
    [
      event.call_id,
      event.status,
      event.sub_status,
      event.call_cost ?? null,
      event.transcript ? JSON.stringify(event.transcript) : null,
    ],
  );
}

Verifying the signature

When a signing secret resolves for the subscription, every delivery carries:

X-Graine-Signature: t=1787649682,v1=8b3c5c4c3fb31d4f0a2e9df5c7d1a6b40e2f8c93a15d7e6b04c8f2a913d5e7b0

• t is the Unix timestamp, in seconds, at which that attempt was signed.
• v1 is a lowercase hex HMAC-SHA256 digest.

The signed material is the timestamp, a literal ., then the exact raw request body bytes:

signed_payload = "{t}" + "." + <raw request body>
signature      = hex(HMAC_SHA256(secret, signed_payload))

The timestamp is inside the signed material specifically so a captured request cannot be replayed with a fresh header: changing t invalidates v1. Reject any delivery whose t is outside a tolerance window — five minutes is a reasonable choice, and comfortably larger than the two-minute retry budget.

Runnable examples

const express = require('express');
const crypto = require('crypto');
 
const app = express();
 
const WEBHOOK_SECRET = process.env.GRAINE_WEBHOOK_SECRET;
const TOLERANCE_SECONDS = 300; // 5 minutes
 
/**
 * Verify an X-Graine-Signature header.
 *
 * @param {Buffer} rawBody  the exact bytes received, before any JSON parsing
 * @param {string} header   the X-Graine-Signature header value
 * @param {string} secret   the subscription's signing secret
 */
function verifyGraineSignature(rawBody, header, secret) {
  if (!header) return false;
 
  // header is "t=<unix_seconds>,v1=<hex>" — parse it without assuming order.
  const parts = Object.fromEntries(
    header.split(',').map((kv) => {
      const i = kv.indexOf('=');
      return [kv.slice(0, i).trim(), kv.slice(i + 1).trim()];
    }),
  );
 
  const timestamp = parts.t;
  const received = parts.v1;
  if (!timestamp || !received) return false;
 
  // Reject replays of an old capture.
  const age = Math.abs(Math.floor(Date.now() / 1000) - Number(timestamp));
  if (!Number.isFinite(age) || age > TOLERANCE_SECONDS) return false;
 
  // signed_payload = "<t>" + "." + <raw body bytes>
  const signedPayload = Buffer.concat([
    Buffer.from(timestamp, 'ascii'),
    Buffer.from('.', 'ascii'),
    rawBody,
  ]);
 
  const expected = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');
 
  // Constant-time compare. timingSafeEqual throws on length mismatch, so guard.
  const a = Buffer.from(expected, 'utf8');
  const b = Buffer.from(received, 'utf8');
  return a.length === b.length && crypto.timingSafeEqual(a, b);
}
 
// `express.json({ verify })` keeps the raw bytes alongside the parsed body.
app.post(
  '/graine/calls',
  express.json({
    limit: '2mb', // the cap is 1,000,000 bytes; leave headroom
    verify: (req, _res, buf) => {
      req.rawBody = buf;
    },
  }),
  async (req, res) => {
    if (!verifyGraineSignature(req.rawBody, req.get('X-Graine-Signature'), WEBHOOK_SECRET)) {
      return res.status(401).send('invalid signature');
    }
 
    const event = req.body;
    console.log(
      'event=%s call=%s status=%s/%s attempt=%s',
      event.event_type,
      event.call_id,
      event.status,
      event.sub_status,
      req.get('X-Graine-Attempt'),
    );
 
    // Acknowledge FIRST, then process. The delivery timeout is 30 seconds and
    // there are only four attempts — never do slow work before responding.
    res.status(200).json({ received: true });
 
    setImmediate(() => {
      enqueueForProcessing(event).catch((err) =>
        console.error('processing failed for %s', event.event_id, err),
      );
    });
  },
);
 
app.listen(4000);

If no signature arrives

The X-Graine-Signature header is absent entirely — not empty, not a placeholder — when no signing secret resolved: a stored secret could not be decrypted (the subscription projection reports secret_unavailable: true in that case), or the organization key could not be minted.

Where to get your secret:

• Dashboard Custom Webhook — open the agent's Custom Webhook dialog and press Reveal next to Signing secret, or call GET /v2/webhooks/signing-secret (requires webhooks:write — holding this value is the ability to forge a delivery to yourself, so a read-only key cannot fetch it). The key is minted on first read and stable afterwards; fetching it never rotates it.
• API subscription — the secret you supplied on POST /v2/webhooks wins for that subscription; without one, the organization key above signs it.

Reject unsigned deliveries in production. If you are receiving them unexpectedly, re-send the secret on the subscription.

Headers on every delivery

Ours are sent first, then your custom headers.

Your custom headers are appended afterwards, minus any whose name is content-type, content-length or host, any beginning x-graine-, and any whose name or value contains a CR or LF. A configured header can therefore never forge our signature or misstate the body length.

Body size and truncation

The body is compact JSON (no spaces after separators), UTF-8, non-ASCII characters left as-is rather than escaped. The maximum body is 1,000,000 bytes.

Nearly every payload is far below that. When one is not — a very long transcript — it is reduced in this order, and each step is applied only if the previous one was not enough:

The same bytes are re-sent on all four attempts, so a body that verified on attempt 1 verifies identically on attempt 4 — only the signature's timestamp differs.

Debugging a missing webhook