Classroom-Based Experimentation in a Digital Learning Platform

Educational research often struggles to balance laboratory-style control with the real-world scale of authentic classrooms. This paper explores how classroom-embedded, digitally integrated experimentation addresses this tension by generating actionable evidence directly within digital learning environments. Using Carnegie Learning’s MATHia intelligent tutoring system alongside UpGrade—an open-source A/B testing platform—the author highlights three diverse field trials encompassing nearly 100 experiments and hundreds of thousands of students. These include an XPRIZE study on localized personalization, an IES-funded reading readability initiative leveraging Large Language Models (LLMs), and an ongoing project evaluating targeted metacognitive prompts. From these large-scale applications, three foundational insights emerge for optimizing digital education research: seamlessly weaving interventions into daily instruction to eliminate teacher burden, capturing real-time learning metrics through intrinsic system-generated data, and prioritizing small, theory-driven manipulations to isolate specific cognitive mechanisms. While logistical constraints like sequence-dependent timing and varying state curricula present ongoing challenges, the paper concludes that embedding rigorous, focused experiments into adaptive software offers a scalable, non-disruptive framework. This approach simultaneously advances learning science theory and drives iterative, evidence-based development in educational technology.