Everyone Did The Course. Nobody Opens The Tool.
I once sat in a review where a client walked through a dashboard showing a 96% course-completion rate for an AI rollout. Everyone nodded. The numbers were green. Then someone from operations asked, almost as an aside, whether anyone on the floor was actually using the tool. The room went quiet, because nobody had measured that, and the honest answer turned out to be no. That gap, between a course that completes and tool adoption in use, is the thing nobody puts on the dashboard. And it’s the only thing the project was ever about.
Completion Measures Attendance, Not Change
A gym membership is a useful way to think about this. The gym can tell you that you scanned your card 11 times last month. It cannot tell you whether you’re any stronger. Checking in is real, it’s easy to count, and it’s almost completely disconnected from the thing you actually signed up for. Course completion is the card scan. A passed quiz is a slightly better one. Neither tells you whether anything changed where the work gets done.
There’s a useful framework for this from a training researcher named Donald Kirkpatrick, who laid out four levels of evaluation. Level 1 is whether people liked the training. Level 2 is whether they learned the material, which a quiz can check. Level 3 (whether the behavior actually changed back on the job) is the one that matters here, and it’s the one almost nobody measures. There’s a level 4 above it too, the actual result the business was after, and the reason level 3 behavior is what I watch is that it’s the leading indicator of those results: fewer callbacks, less rework, fewer of the exact mistakes the tool was bought to stop. When the behavior changes on the job, those floor numbers follow. You don’t need the academic vocabulary to run this well. You just need to know that completion and quiz scores live at the easy levels, and the thing you care about lives a level up where it’s harder to see.
What Adoption Actually Looks Like
Behavior change leaves traces if you know where to look. Is the tool getting opened at the workflow step it was meant for, weeks after the training ended, without anyone reminding people? Are workers bringing it up on their own, asking about edge cases, telling you where it falls short? Those questions only come from people who are actually using something. When the people you trained start telling you where the tool falls down, it can sound like a complaint, but what you’re really hearing is people who’ve folded the thing into their day far enough to find its edges. That’s the sound of adoption.
Usage that holds past the first two weeks is the signal I trust most. Almost anything gets a bump right after training. The question is whether the curve flattens out at a real number or drops back to zero once the novelty and the manager’s attention move on.
Instrument It Without Turning Into Surveillance
The wrong way to measure adoption is to start logging every keystroke and tracking individual workers against each other. The moment people feel watched, you’ve poisoned the thing you’re trying to measure, and you’ve earned yourself a floor full of people who’ll use the tool only when they think someone’s looking.
Aggregate signals are enough, and most of them are countable without singling anyone out. How often the tool gets opened at the right workflow step, counted by team rather than by name. Whether the support tickets or callbacks the tool was meant to cut are actually trending down over a quarter. A short, honest check-in a month out that asks what’s working and what isn’t. A quiet word with a couple of frontline leads about whether they’re seeing the behavior in the wild. Every one of those reads at the team level, never the individual one. You’re trying to learn whether the work changed, not to build a case file on any one person. The lighter your hand, the truer the reading.
Set This Expectation Before The Work Starts
Here is the part that has to happen before the engagement, not after. When a buyer asks me to build AI training, completion is usually the number they have in their head as the finish line. So I tell them early that completion isn’t the deliverable, behavior change is, and that we’ll be looking at usage on the floor 60 and 90 days out, not just whether everyone clicked through by the deadline. Some habits settle in by then and some take longer, so I treat those checkpoints as a reading on the trend towards adoption (or otherwise) rather than a final grade.
That conversation changes the whole shape of the project, because a course built to be completed and a course built to be used are not the same course. A few buyers push back at first, because completion is the number they came in expecting to report upward. But most, once they sit with it, would rather know the truth than carry a 96% completion rate that hid a tool nobody touched. They wanted the tool used. They just hadn’t been told that completion and use were different things they had to choose between at the start. So I tell them, and we measure the right one from day one.
