What Does Learning Look Like in 12 Months?
I have kids and I have no idea what their school is going to look like in 12 years.
Hell, nobody in edtech knows what school looks like 12 months from now. The ground keeps moving ever since LLMs started doing everyone's homework. So I stopped guessing and started building from a field that already solved the hard part: Intelligence Studies and reasoning from sources you can't fully trust.
That is intelligence work. OSINT, source verification, structured analysis. The Intelligence Community has spent decades on the exact problem the rest of us just got handed. "The inputs are unreliable, now what?"
That's the bet behind OSINTstitute Academy.
LLMs hallucinate, the web is half-bot (soon to be all bot?), screenshots are fake - verification is the new core skill. Here are the givens I am building around:
The information environment is permanently unreliable now (LLMs, bot web, fabrications). Steady state, not phase.
Humans still have to act in it. Form beliefs, make calls, eat consequences.
AI commoditizes everything that reduces to pattern-matching against existing data - including most of what current "AI literacy" courses teach.
The IC's actual contribution isn't "how to know the truth" (that's epistemics, not their lane). It's narrower and more useful: how to act on incomplete, contradictory, possibly-deceptive information without paralysis or overconfidence.
AI doesn't bear consequences for being wrong. You do. That asymmetry is the entire reason humans still need a process - not "AI literacy," but a way to make defensible calls when the inputs are unreliable.
What's already been done
Before I get too high on my own supply: calibration as a concept isn't new. LLM training research has been working on it for years (RLHF, abstention rewards, "models mostly know what they know"). Forecasting communities like Tetlock's Good Judgment Project and Metaculus have run on Brier scores for over a decade and produced measurable superforecasters. The IC has trained analysts on this stuff for a century. Decision-science academia has the textbooks. The pieces all exist.
What doesn't exist - and what I'm betting nobody else is going to ship in time - is the combination: IC tradecraft (broad verification, not just future-event forecasting) + Brier-style calibration scoring + consumer packaging + AI-saturation framing, in one product, aimed at people who aren't already analysts or quants. Model-side calibration is for the model, not the user. Forecasting communities train a narrow surface. Professional pipelines are gated. Edtech "AI literacy" courses are teaching prompt engineering and bias awareness and calling it a day. The four ingredients are sitting in four separate kitchens.
The thesis in one sentence
The durable skill of the next decade isn't prompting, tool fluency, or "media literacy" - it's running your own commit → calibrate → update loop, formalized over a century by intelligence professionals.
Three hunches I'm testing with this platform
A graduate is someone who can be wrong on purpose.
Willing to state a belief at a confidence level, knowing they might be wrong, because that's the only way feedback can improve calibration. The fence-sitter (refuses to commit, defers to AI) and the dogmatist (commits and refuses to update) are the failure modes. The graduate sits between.
Spy tradecraft transfers better to "AI literacy" than anything currently shipping under that name - Prompting is useful. Tool fluency is useful. But the durable skill is verification under uncertainty.
A course written today can't answer a question from October. The pipeline has to assume it'll be stale - The pipeline has to expect staleness. Courses need to be revisable, evidence-linked, and easy to update when the information environment changes. Quickly.
I expect to be wrong about a lot, and that is the point. I'm just one guy hoping not to get eaten by AI while hoping I only show the right thing to the dark forest. So I will adhere to the wisdom of Ward Cunningham and post the "wrong thing" to the internet then watch the right answers come in... nicely or otherwise, and use the feedback to make the product less wrong.
Platform is live at https://www.osintstitute.com/ - 12 starting courses, ~240 lessons, handful of paying customers. Small numbers right now, but a real bet costing me many hours away from those kids.
Go ahead, tell me how I am wrong. That's the point.
I like the idea that the real skill is making defensible decisions under uncertainty. It reminds me of cybersecurity, where tools like "Network Threat Detection" are only effective if people know how to interpret the evidence rather than blindly trusting every alert.
It is incredibly daunting trying to figure out how to prepare the next generation for a world where the line between a verified fact and a hallucinated bot response has completely disappeared.
The real shift here is recognizing that the "Intelligence Community" model works because it focuses on the weight of evidence rather than the search for a perfect truth, which is the only way to avoid paralysis when your data source is an unreliable LLM.
Do you think that teaching people how to assign "confidence scores" to their own beliefs will be the hardest part of the curriculum for those who are used to the binary right-or-wrong nature of traditional schooling?
I'd bet it's not the hardest part, people adjust to "you can be 60% right" pretty fast once they see a score like that a few times. The sneakier, harder thing: getting people to commit to their own authority instead of having AI sit as the new "permission-to-speak" machine replacing a teacher or a textbook.
Your 'weight of evidence vs. perfect truth' line is sharper than what I wrote in the post, by the way.
Breaking the habit of using AI as a "permission-to-speak" machine is the real hurdle because once you outsource your judgment to a model, you lose the ability to defend your own conclusions.
Moving from "is this true?" to "how much evidence supports this?" is the only way to stay functional in an information environment that is permanently noisy and often deceptive.
I apply this exact principle of defensible calls in my high-tier PR and media placement work where we use verified data to build undeniable authority for brands on major news outlets.
Since the goal is to have users commit to their own authority, does the platform include a "post-mortem" feature where they can analyze why their confidence score was off after a fact is eventually verified?