Using The Wrong Yardsticks In The Age Of AΙ
There’s a painfully familiar corporate training experience where an employee completes a course consisting of polished slides, predictable interactions, and a handful of recall questions. They pass the assessment, get a completion certificate, and return to work without necessarily developing a new capability. Instructional designer Beth Pennington, M.S., describes a Six Sigma White Belt course she took through an AI-powered platform: she expected structured, hands-on instruction with real-world applications, but instead, encountered what she described as a dull, AI-generated slideshow with zero depth and zero value. [1]
Generative AI can now help organizations produce more courses faster, but faster production alone doesn’t address longstanding problems with irrelevant content, superficial assessments, and limited workplace application. AI didn’t invent bad eLearning, but it did make bad eLearning easier to produce.
Faster Doesn’t Mean Better
Are organizations using AI to improve learning or simply to manufacture more training?
Generative AI offers genuine advantages in the speed of instructional development. AI-assisted authoring can accelerate scripting, storyboarding, assessment writing, content adaptation, and media production. These capabilities can reduce development costs, shorten turnaround times, and potentially free instructional designers to concentrate on more complex work. A randomized controlled trial conducted by the Education Endowment Foundation found teachers using ChatGPT spent significantly less time preparing lessons and instructional resources than teachers who did not use generative AI, demonstrating its potential to reduce development workloads. [2]
However, reducing the time required to produce instructional materials is not the same as demonstrating those materials improve learning or workplace performance. A course generated in 30 minutes isn’t necessarily worse than one developed over six weeks. It also isn’t necessarily better. The problem arises when speed and volume become proxies for quality.
What Are We Measuring? Output Vs. Impact
The concern isn’t simply the technology. It’s the organizational incentives that determine how it’s used. The ease of producing training raises a familiar question for L&D: What are we actually measuring? Course production tells us how much content we’ve created and how efficiently we’ve created it. Participation metrics tell us how many employees enrolled or completed a module. Learning assessments can tell us whether employees acquired the intended knowledge or skills, while workplace performance measures examine whether they’re applying those capabilities on the job. Business outcomes take the evaluation further, asking whether those changes contributed to meaningful organizational results, such as fewer errors, improved productivity, or better customer experiences.
Each measure serves a purpose, but they are not interchangeable. According to a 2025 Association for Talent Development (ATD) survey of 277 talent development professionals, 93% of organizations collect participant satisfaction data, but only 30% report being good at using learning program data to inform business decisions. [3] A completed course is evidence of participation, not proof of proficiency.
AI doesn’t create the incentive problem, but it can amplify it. If L&D is rewarded primarily for how quickly it delivers courses and how many people complete them, then AI makes it easier to optimize those metrics without necessarily improving organizational performance.
Informing Is Not Teaching
Generating accurate content isn’t equivalent to designing an effective learning experience; it is informing, but it’s not teaching. An AI system can produce a coherent explanation of a workplace procedure, generate examples, and write five knowledge-check questions. While those may all be useful components of instruction, recalling information is insufficient when the learning objective requires employees to demonstrate a skill or make sound decisions.
Effective instruction begins with relevance: identifying the actual performance need and ensuring the learning experience addresses it. It requires practice, giving employees opportunities to make decisions, solve problems, or perform tasks that resemble their real work. Meaningful feedback helps learners recognize mistakes, understand their consequences, and improve their performance. And ultimately, application matters: employees need opportunities and support to transfer what they’ve learned into their daily responsibilities.
Research reinforces these principles. A 2013 review by Dunlosky and colleagues identified practice testing and distributed practice as particularly effective techniques for improving learning and retention. [4] Hattie and Timperley’s 2007 review found that feedback can substantially influence learning, although its effectiveness depends on the type of feedback and how it is delivered. [5]
For example, AI might generate a workplace safety course with a plausible scenario about hazardous conditions, but its instructional value depends on whether the scenario reflects actual risks, requires meaningful decisions, and provides feedback that helps prevent errors. Even well-designed instruction, however, doesn’t guarantee workplace application. A meta-analysis of 89 empirical studies by Blume and colleagues found training transfer (i.e., the application of learned skills to new tasks and contexts) is positively associated with learner characteristics such as motivation and cognitive ability, as well as a supportive work environment. [6]
Human-designed courses can fail on every one of these dimensions, too. AI can make a course look finished long before anyone has established it will work.
AI In Sound Instructional Design
Organizations should use AI to strengthen the learning process, not simply accelerate content generation. Instead of starting with “What course can we create?”, start with “What must employees be able to do differently?” And then, “Is training the appropriate intervention?”
If so, then AI can help develop realistic scenarios, generate varied practice opportunities, support individualized feedback, adapt instruction to learners’ existing knowledge, and analyze learning data. But organizations still need to validate those experiences against actual performance requirements and evaluate whether they produce the intended results.
In a 2025 experimental study involving 447 participants, Netland and colleagues found no statistically significant difference in knowledge assessment scores between viewers of AI-generated and human-produced instructional videos, although participants rated the human-produced videos more favorably. [7] A 2025 rapid review of 15 studies on AI-generated instructional videos in higher education, which included the Netland study, reported comparable short-term learning outcomes in several experimental comparisons. [8] However, the review emphasized the educational value of these videos depends on instructional design and how the technology is integrated into the learning experience.
These findings suggest the critical distinction isn’t necessarily between AI-generated and human-produced instruction. It’s between instruction designed to develop capability and content produced simply because it can be produced.
A completed course, a passing score, and a certificate are easy to count. Whether the employee can perform more effectively afterwards is harder to establish, and considerably more important. AI has made it easier than ever to produce training, an achievement worth recognizing, but organizations must decide what they’re trying to accelerate. The future of eLearning shouldn’t be measured in courses produced, but rather in capabilities developed.
References:
[1] AI in Corporate Training: A Revolution Gone Wrong
[2] ChatGPT in Lesson Preparation: A Teacher Choices Trial
[3] Organizations Struggle With Measuring the Impact of Training
[4] Improving Students’ Learning With Effective Learning Techniques
[6] Transfer of Training: A Meta-Analytic Review
[7] Comparing human-made and AI-generated teaching videos
[8] A rapid review of using AI-generated instructional videos in higher education
