Hookpilot

Funny thing happened last week. I run Hookpilot, a webhook reliability platform, so my entire pitch to customers is "stop trusting webhook delivery order, treat every event as unreliable, build for that." Then I went and got bitten by the exact thing I warn people about, on my own billing system, in production.

Here's what happened.

A user cancels their subscription. Paddle (our payment processor) defers it to end of billing period, like most processors do. Nothing charges again, access continues until the date, then it lapses. Standard stuff. Except our own reminder email didn't know the difference between "will renew" and "about to end," so a customer who'd already cancelled got an email telling them their subscription renews soon. Not a great look for a company selling reliability.

So I fixed it. Added a field that tracks the pending cancellation, wired it into the webhook handler, built a proper "keep my plan" button, shipped it, tested it, felt good about it.

Then I tried it on my own account and it didn't work. The field stayed empty even though I'd genuinely cancelled.

I could've guessed at a dozen causes. Instead I pulled the actual webhook delivery logs from Paddle's dashboard. Turned out Paddle had fired two subscription.updated events for one cancellation, 185 milliseconds apart. One carried the cancellation details, one didn't. My server processed them, and the one without the cancellation data landed second, wiping out the correct state the first one had just set.

Nothing was wrong with either webhook. Paddle doesn't promise they'll arrive in the order they happened, and mine hadn't. Whichever one my server happened to process last, won, regardless of which one was actually more recent.

That's the exact scenario I built Hookpilot to protect other people's endpoints from. I'd just never applied the lesson to my own backend, because "my own billing system" didn't feel like the kind of place that needed the same paranoia as customer-facing webhook infrastructure. It was.

The fix was small once I knew what to look for: stamp every applied event with its own timestamp, and refuse to let an older one overwrite a newer one. A few lines of code. The part that actually mattered was realizing delivery order isn't something you get to assume anywhere, including in your own house.

If you're running anything that consumes webhooks, and you're not already checking whether the event you're processing is actually the newest thing that happened, this is your reminder. It won't be obvious until it shows up on your billing page, or worse, until something a customer notices before you do.

Hookpilot exists so other people don't have to learn this the hard way. I still did.