WebhookNotDelivering

What a missing delivery means

A GitHub webhook that "stopped working" has almost always failed in one of four ways: your endpoint returned a non-2xx code, it took longer than 10 seconds to respond, the signature did not validate so your handler rejected it, or a firewall blocked GitHub's sending IPs. GitHub records every attempt — start at Settings > Webhooks > your hook > Recent deliveries, which lists deliveries from the past 3 days with their response code and body (Redelivering webhooks). That page tells you whether GitHub sent and failed (your problem) or never sent (an event/config problem) before you touch any code.

One thing to internalize up front: GitHub does not automatically redeliver failed deliveries (Handling failed webhook deliveries). A transient outage on your side means those events are gone unless you redeliver them.

Why it happens

GitHub's contract is strict and well documented: "Your server should respond with a 2XX response within 10 seconds of receiving a webhook delivery. If your server takes longer than that to respond, then GitHub terminates the connection and considers the delivery a failure" (Best practices for using webhooks). Because GitHub expects a 2xx, any non-2xx response — a 4xx or 5xx from your handler — leaves the delivery without a successful status, which you redeliver manually since GitHub does not auto-retry (Handling failed webhook deliveries).

The other recurring causes are environmental: GitHub's sending IPs are not allow-listed by your firewall, your TLS chain is incomplete so GitHub's client rejects the handshake, or the payload exceeded the 25 MB cap and GitHub declined to deliver it (Webhook events and payloads).

Diagnose it

1. Read Recent Deliveries. Open the hook and look at the response column. A red entry with a 4xx/5xx body is your handler erroring. A timeout entry means you exceeded 10 seconds. No entry for an event you expected means GitHub never sent it — the event is not subscribed, or the hook is on the wrong scope (repo vs org vs app).

2. Pull deliveries via API if you want them in a terminal or a script. The repository-webhook endpoints are:

# List recent deliveries (needs read:repo_hook or repo scope).
gh api repos/{owner}/{repo}/hooks/{hook_id}/deliveries \
  --jq '.[] | {id, event, status_code: .status_code, delivered_at}'

# Inspect one delivery in full (request + response).
gh api repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}

# Redeliver it (a fresh attempt).
gh api -X POST repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}/attempts

Those paths are the documented repository-webhook delivery endpoints (Repository webhooks REST API); org and app webhooks have parallel endpoints under /orgs/{org}/hooks/... and /app/hook/deliveries.

3. Confirm GitHub can reach you. If deliveries show as failed with connection errors, verify your firewall allows GitHub's sending IPs. The current set is in the hooks key of the meta endpoint:

# The hooks array is the IP ranges GitHub sends webhooks from.
gh api meta --jq '.hooks'

GitHub's own guidance: "make sure that your server allows connections from GitHub's IP addresses. You can use the GET /meta endpoint to find the current list" (Troubleshooting webhooks). For TLS errors ("Peer certificate cannot be authenticated"), check your chain with an external SSL test or openssl s_client — the cause is a self-signed cert or a missing intermediate (Troubleshooting webhooks).

Root causes

• Slow handler — synchronous work (DB writes, downstream API calls) inside the request pushes you past 10 seconds. The fix is to acknowledge immediately and process asynchronously.
• Handler erroring — an exception returns 5xx; a routing or auth mistake returns 4xx. The delivery body in Recent Deliveries shows your own error.
• Signature rejection — you validate the wrong header or algorithm (see below).
• Firewall / IP — GitHub's sending IPs are blocked at your perimeter.
• TLS chain — incomplete certificate chain rejected by GitHub's client.
• Oversized payload — over 25 MB, so GitHub sends nothing for that event.

Signature validation (the most common self-inflicted failure)

GitHub signs each delivery so you can verify it came from GitHub. Validate the X-Hub-Signature-256 header, which is an HMAC-SHA256 of the raw request body keyed with your webhook secret, prefixed sha256= (Validating webhook deliveries). The older X-Hub-Signature header is HMAC-SHA1 and "only included for legacy purposes" — do not build on it.

import hmac, hashlib

def is_valid(payload_body: bytes, secret: str, header: str) -> bool:
    if not header:
        return False
    expected = "sha256=" + hmac.new(
        secret.encode(), payload_body, hashlib.sha256
    ).hexdigest()
    # constant-time compare; never use ==
    return hmac.compare_digest(expected, header)

Two things bite people here: signing the parsed body instead of the raw bytes (any re-serialization changes the HMAC), and using == instead of a constant-time compare. GitHub's examples reject a mismatch with HTTP 403 "Request signatures didn't match!" (Validating webhook deliveries).

Resolution and recommended approach (with tradeoffs)

Acknowledge fast, process later. Have the endpoint validate the signature, enqueue the raw event, return 200 immediately, and do the real work off the request path. This is the single change that eliminates the timeout class of failures. The tradeoff is you now own a queue and its retry/idempotency semantics — but that is the correct place for that complexity, not inside a 10-second budget.

Make handlers idempotent. Because you can (and on transient failures, will) redeliver, and GitHub may send retries, design handlers to tolerate the same event twice. Key on the delivery GUID. The cost is a small dedupe store; the alternative is duplicated side effects.

Replay, don't despair. When you fix a handler bug, redeliver the failed deliveries from the past 3 days rather than waiting for new events — but note the 3-day retention window: anything older is not replayable from GitHub (Redelivering webhooks).

Validation steps

After a fix, redeliver a known-failed delivery and confirm a 2xx:

gh api -X POST repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id}/attempts
gh api repos/{owner}/{repo}/hooks/{hook_id}/deliveries/{delivery_id} --jq '.status_code'
# want: 200 (or any 2xx)

Then trigger a real event and confirm a fresh green entry appears in Recent Deliveries within seconds.

Prevention strategies

• Allow-list from /meta, and re-pull it periodically — GitHub's hook IPs change, and a stale allow list silently drops deliveries.
• Alert on delivery failure, not just on processing errors. A red Recent Delivery is the earliest signal; if you only watch your queue, you miss the events GitHub could not hand you.
• Keep payloads under the cap by subscribing to the narrowest event set you need; do not subscribe to everything and filter, which also raises your odds of hitting the 25 MB limit on bulk pushes.

Operational lessons

The deceptive part of this failure is that GitHub looks fine — the event happened, the run is green, the PR updated — and yet your system never heard about it. The asymmetry (GitHub does not retry, deliveries expire after 3 days) means a 20-minute outage on your receiver can quietly drop a day's worth of events, and you only notice when downstream state is stale. Treat the receiver as a system that must never 5xx-and-forget: validate, enqueue, ack, replay.

For a platform that reconstructs the change behind an incident, a dropped push or pull_request webhook is a hole in the timeline — the commit that caused the problem is invisible until someone notices the gap. That is why the durable pattern is acknowledge-fast plus idempotent replay: it makes the event stream complete and re-runnable, which is the prerequisite for correlating a deploy to the alert it caused.