Picked by rule: the highest evidence levels first (studies with real learners), then newest.
Level 3: Controlled comparisonLearners: Adult learnersOutcome: Learning or achievementPreprintPreregisteredShort exposureStudy type: Randomized controlled trial
2026-09-26 · arXiv (back catalog) · Study (preprint)
Chatbots deliver real-time fact-checks, but whether a chatbot correction leaves anything behind once the chatbot is gone - metalearning, distinct from correcting misbeliefs - is untested. We report a preregistered, three-country randomized experiment (USA, India, Singapore; N ~ 2,200) on out-of-context image...
Why this label for Chatbot Engagement Does Not Always Beget Metalearning: Evidence from Three Count
- Level 3: Controlled comparison: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 randomized participants across four conditions with a one-week unaided discernment retest; R7 preprint cap.
- Grader B: R5: preregistered randomized experiment with N~2,200 real adults and a discernment (learning) outcome at one-week retest gives Level 3; R7 preprint and single-session caps hold it at 3, and as a preprint it cannot be a Level-4 candidate.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 3: Controlled comparisonLearners: Higher-ed studentsOutcome: Learning or achievementPreprintStudy type: Quasi-experimental
2026-07-23 · arXiv (back catalog) · Study (preprint)
The availability of generative AI (GenAI) has raised concerns that college students substitute GenAI for cognitive effort, earning high grades without learning. If this "GenAI substitution hypothesis" is true, grades should rise disproportionately in GenAI-susceptible courses--those relying more on assessments like...
Why this label for Generative AI Availability, Grades, and Student Satisfaction at a Large Universi
- Level 3: Controlled comparison: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: the difference-in-differences analysis compares courses before and after AI availability using baseline administrative data.
- Grader B: R5 design ceiling: difference-in-differences quasi-experiment on administrative data (138,386 students) with a comparison and a pre-period gives Level 3; R7 preprint cap is 3; grades are a proxy for learning, so review for R6 indirectness
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Cognitive load: matched 'cognitive effort'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 3: Controlled comparisonLevel-4 candidate (held at 3: preprint, needs full-text human check)Outcome: Learning or achievementPreprintStudy type: Randomized controlled trial
2026-04-06 · arXiv (back catalog) · Study (preprint)
People often optimize for long-term goals in collaboration: A mentor or companion doesn't just answer questions, but also scaffolds learning, tracks progress, and prioritizes the other person's growth over immediate results. In contrast, current AI systems are fundamentally short-sighted collaborators -- optimized for...
Why this label for AI Assistance Reduces Persistence and Hurts Independent Performance
- Level 3: Controlled comparison (human override): Label correction only, level unchanged. The arXiv abstract describes participants only as 'people' and does not say how long they used the AI, so the automated labels 'Learners: Adult learners' and 'Short exposure' are not supported by the source. Removed to match the corrected explainer.
- Changed from: Level 3: Controlled comparison
- Grader A: R5 design ceiling: randomized controlled trials assign human participants and measure unassisted performance; Level 4 verification and uncertainty information are not reported.
- Grader B: R5 design: series of RCTs (N=1,222) on unassisted performance and persistence; R7 preprint and short-exposure caps hold at max Level 3; human review needed before any upgrade.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Cognitive load: matched 'cognitive offloading'
- Reading and writing: matched 'reading comprehension'
Graded blind by two AI graders against our published method. Human grade change 2026-10-24, approved by Matt, Oct 24 2026 Method
Level 2: CorrelationalStudy type: Quasi-experimental
2026-10-19 · Frontiers in Education · Study
Effective communication is fundamental to high-quality healthcare. Communication skills training is a core component of medical curricula, yet opportunities to practice with standardized patients (SPs) are limited, costly, and can elicit anxiety in novice learners.
Why this label for Use of a GenAI-patient for medical communication training for undergraduate medi
- Level 2: Quasi-experiment, n=383, groups allocated by class schedule; outcomes are self-reported nervousness, preparedness and self-efficacy, so the perception-only cap holds it at Level 2.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Level 2: CorrelationalLearners: Higher-ed studentsOutcome: Behavior or engagementPreprintPreregisteredPerception onlyStudy type: Cohort / case-control
2026-10-07 · arXiv (back catalog) · Study (preprint)
AI learning tools are rapidly entering classrooms, but evidence about whether they help students learn is mixed and rests mostly on test scores. Comparatively less research addresses whether the use of AI changes students' live learning behaviors in class.
Why this label for How assigned AI use before class shapes active student engagement in class
- Level 2: Correlational: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R7 hard cap: randomized assignment is reported, but the measured outcome is classroom engagement rather than learning.
- Grader B: R7 engagement-only cap: a preregistered randomized field experiment (n=759 MBA students) that would otherwise be Level 3, but outcomes are class contributions and self-reported perceptions, with no learning measure, so max 2
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Motivation and engagement: matched 'student engagement'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
