A directional error on your newest data point manufactures a trend

Last time I posted here it was about a filter that made a live source look dead. Same shape of problem again, one layer up. This time the instrument was fine and the calendar was the bug.

I build devdigest, a daily tech digest. Every change that changes behaviour gets reviewed against real data five days later, and I write the numbers down before I let myself have an opinion about them.

This week's series, the share of each digest landing in the four categories a fix I had just shipped was meant to feed:

69.2, 67.6, 60.0, 53.0

Four days, straight down. I had the write-up half composed in my head.

Then I noticed the last number was wrong.

Subscribers choose when their digest arrives. The first goes out just after five in the morning, the last at midday. My measurement treated the current day as a finished one, so reading it in the morning meant measuring a day that had not happened yet. The real figure was 53.9, not 53.0.

Nine tenths of a point. Not exciting. Here is why it is worth writing down.

The error is directional, not random. The late digests skew one way, so an early reading is always wrong in the same direction. And it is always wrong on the newest point, which is exactly where you look when you are deciding whether something is trending.

A random error adds noise you can see. A directional error on the most recent point manufactures a slope.

Day five then came in at 61.9 and did not continue the decline.

My first instinct was to write that up as the decline never having been real. That instinct is wrong, and it is wrong in the same way as the one it replaced. Four points looked like a trend. Five points look like a recovery. Neither is something five points can tell you, and day six could resume the fall.

What I can say is narrower. The four-day drop was 15.3 points. The largest four-day drop across the twenty-two days before it was 8.7. So something real happened, and waving it away as noise would have been the mirror image of the mistake I had just caught myself making.

The measurement error and the reading error turn out to be the same error wearing different clothes: wanting a small number of days to settle a question they cannot settle.

So I am not calling it. There is a date in my notes to read the same series over eleven days instead of five, and I will post what it says.

The rule now sitting in the code: a partial day does not look like missing data. It looks like a data point, and you will believe it.

If anything you rely on has a "today" column, go and find out what time today finishes.

devdigest is one tech email a day from more than a hundred sources, no ads, nothing sold on. 14 days free, no card: https://devdigest.io/?ref=indiehackers