Level 2: CorrelationalLearners: K-12 studentsOutcome: Behavior or engagementPreprintStudy type: Cohort / case-control
2026-10-06 · arXiv (back catalog) · Study (preprint)
Algorithmic recommendations can help participants navigate large matching markets. For example, recommendations for school and college choices may reduce information frictions and disparities in access to high-performing programs.
Why this label for Personalized Recommendations Without Inducing Congestion: Mitigating Disparities
- Level 2: Correlational: Change to 2. Randomized deployment in the NYC high school match; the outcome is school-choice behavior, not learning, so held at 2 after final review.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: applicants were assigned in a randomized trial, with ranking and matching behavior measured; Level 4 learning-outcome criteria are unmet.
- Grader B: R7 behavior/engagement-only cap: an RCT in the NYC high school match with real applicants, but outcomes are ranking and matching behavior, not learning, so capped at 2; preliminary conference version, treated as a preprint
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 2: CorrelationalLearners: Adults not in a learning role (raters)Outcome: Behavior or engagementPreprintStudy type: Cohort / case-control
2026-09-18 · arXiv (back catalog) · Study (preprint)
AI is rapidly transforming higher education, including the application process, yet relatively little is known about its use and consequences. To help close this gap, we analyze nearly 7{,}500 applications submitted between 2020 and 2025 to a large public policy master's program in the United States.
Why this label for AI-written admissions essays are widespread but penalized
- Level 2: Correlational: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 observational comparisons of applicants and an admissions-officer rating experiment do not establish learner assignment or baseline equivalence.
- Grader B: R5: an observational analysis of ~7,500 applications with comparable-applicant controls and a ChatGPT-introduction natural experiment is correlational (Level 2); the outcomes are essay quality and admission, not learning. The admissions-officer arm is a rater study.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 2: CorrelationalLearners: Higher-ed studentsOutcome: Learning or achievementPreprintSmall or impreciseStudy type: Cohort / case-control
2026-06-21 · arXiv (back catalog) · Study (preprint)
Pretesting - attempting problems before instruction - supports learning by activating prior knowledge and sharpening attention to subsequent instruction. Recent work suggests that adaptive AI-assisted pretesting can yield further advantages, particularly for tasks requiring higher-order reasoning, yet it remains...
Why this label for Do Gains from Generative AI-Enabled Adaptive Pretesting Persist? Evidence from a
- Level 2: Correlational: Randomized retention study, but sample size not reported; held at 2 until n is verified.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: undergraduates were randomly assigned to study conditions and assessed on posttest performance.
- Grader B: R5 RCT (undergraduates randomly assigned, seven-week retention, learning outcome) gives Level 3; R6 imprecision lowers it to 2 because n and intervals are not reported (RCT without n, needs human review)
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Retrieval practice: matched 'retrieval practice'
- Spacing and interleaving: matched 'spaced retrieval'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 2: CorrelationalProvisional (rules only)
2026-06-04 · ERIC (peer reviewed) · Study
Dialogic peer feedback in synchronous online discussions provides multiple educational benefits; however, its implementation is often constrained by students' cognitive and interactional limitations. To address this issue, this study investigated the effects of argumentation scaffold and role assignment on students'...
Why this label for Fostering Dialogic Peer Feedback in Synchronous Online Discussions: The Effect o
- Level 2: Correlational: provisional rules-only grade, pending grader consensus and review: R5 design: comparison without baseline (F5): Comparison groups but no baseline equivalence reported.
- Feedback: matched 'Peer Feedback'
- Metacognition and self-regulation: matched 'metacognition'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryStudy type: Cross-sectional
2026-10-19 · Frontiers in Education · Study
The advent of generative artificial intelligence (AI) is transforming academic production and confronting the social sciences with the challenge of preserving critical thinking in the face of mechanically generated content. This research examines the perceptions and practices of social science lecturers and students...
Why this label for Perceptions, practices and governance of generative AI in university social scie
- Level 1: Mixed-methods: interviews, non-probabilistic survey of 200 students, policy content analysis. Perception outcomes.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-10-18 · arXiv physics.ed-ph · Study (preprint)
In the "chair challenge" a person stands several foot lengths from a wall, bends forward until the head touches the wall, lifts a chair and tries to stand up. Online videos suggest that women can do this and men cannot, supposedly because women have a lower centre of mass or men have larger feet.
Why this label for The chair challenge: can mechanics really tell men from women? A centre-of-mass
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryStudy type: Case study / case series
2026-10-05 · arXiv search: AI + education · Study (preprint)
Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes...
Why this label for Responsible Institutional Analytics: Interpreting Bias with AI Support
- Level 1: Qualitative study with stakeholders on four IA cases.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-10-02 · arXiv physics.ed-ph · Study (preprint)
Public Exams are an assessment system designed in the context of Biology Education Research to increase student engagement in the exam design process. Students are shown incomplete versions of test questions ahead of time, with the format of the question revealed but some details withheld.
Why this label for Adapting Public Exams for Introductory Physics: A Mixed-Methods Investigation of
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
- Motivation and engagement: matched 'student engagement'
- Assessment and grading: matched 'Exams'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Model performancePreprintSmall or imprecise
2026-09-29 · arXiv (back catalog) · Study (preprint)
AI-assisted cheating has become an important threat to the security of online exams. This study examines whether the risk of AI-assisted cheating in the final exam can be predicted using students' digital traces in the learning management system (LMS) during the first eight weeks of the semester.
Why this label for Early Prediction of AI-Assisted Cheating Risk in Online Exams Through Learning A
- Level 1: Descriptive or exploratory: Change to 1. Predictive analysis of LMS logs from 52 students; correlational design downgraded once for imprecision (n under 100, no interval).
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R4 real-learner gate: student logs train a cheating-risk predictor, and predictive accuracy is the reported outcome.
- Grader B: R5 predictive/correlational design (Level 2), downgraded under R6 imprecision: n=52 with no interval information
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Learning or achievementPreprintResearcher-made measureStudy type: Case study / case series
2026-09-28 · arXiv (back catalog) · Study (preprint)
The rapid proliferation of large language models (LLMs) in the context of education has introduced significant challenges in enforcement of academic integrity, especially in programming courses. We present Argus, an automated detection system for LLM-assisted student work in undergraduate C programming assignments.
Why this label for Argus: Academic Integrity in the Era of Generative AI
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: retrospective flagged-use and exam-performance findings report correlation without statistical controls.
- Grader B: R5 design ceiling: system description plus a simple correlation between detector flags and exam scores, with no statistical controls reported, so Level 1; the detector measure is unvalidated
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Case study / case series
2026-09-28 · arXiv (back catalog) · Study (preprint)
Academic access programs distribute frontier artificial intelligence as research infrastructure, but awards and institutional authorization are distinct stages. We examine the handoff through structured coding of 15 publicly documented access pathways captured on 19 September 2026, of which ten met prespecified...
Why this label for After the Award: The Authorization Gap in Academic Access to Frontier AI
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 descriptive process trace of access programs and one institution, with no learner outcome.
- Grader B: R5 design ceiling: descriptive document coding of 10 access programs plus a single-institution process trace; no learners or comparison group, so Level 1 (descriptive).
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-09-27 · arXiv physics.ed-ph · Study (preprint)
The African School of Fundamental Physics and Applications, also known as the African School of Physics (ASP), was initiated in 2010, as a three-week biennial event, to offer additional training in fundamental and applied physics to African students with a minimum of three-year university education. Since its...
Why this label for Activity Report on the Ninth African School of Fundamental Physics and Applicati
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Behavior or engagementPreprintPerception onlyStudy type: Case study / case series
2026-09-22 · arXiv (back catalog) · Study (preprint)
This paper presents a multi-source classroom study conducted during a 10-week quarter in data science courses at Drexel University. We first investigate the behaviors of student engagement with large language models (LLMs) using four surveys across three data science courses.
Why this label for Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data S
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 classroom surveys describe student use and perceptions; response data also test LLM question ratings without a learner comparison.
- Grader B: R5: surveys and an item-difficulty correlational analysis in a 10-week course with no comparison group are descriptive (Level 1); the R7 perception/engagement cap also applies to learning claims.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Retrieval practice: matched 'retrieval practice'
- Motivation and engagement: matched 'student engagement'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Learning or achievementPreprintSmall or impreciseShort exposureResearcher-made measureStudy type: Case study / case series
2026-09-18 · arXiv (back catalog) · Study (preprint)
Generative AI can improve students' programming performance, but successful task completion may not reflect what they retain. We examined performance, retention, cognitive load, and ownership in a controlled between-subjects experiment with 59 undergraduate computer science students, 55 were retained for analysis.
Why this label for Your Programming Students' Cognition with ChatGPT: Higher Performance, Lower Ret
- Level 1: Descriptive or exploratory: Between-subjects experiment, 55 analyzed, no baseline equivalence and no interval; downgraded to 1.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 controlled comparison lacks explicit random assignment or baseline data; R6 imprecision lowers Level 2 for 55 analyzed students without an interval or power report.
- Grader B: R5: a controlled between-subjects experiment with a recall (learning) outcome is Level 3; R6 imprecision (n=55, no intervals) lowers it to Level 2. R7 short-exposure cap also noted.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Cognitive load: matched 'cognitive load'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Perception or attitudePeer-reviewed (journal reference listed)Perception onlySmall or impreciseStudy type: Cross-sectional
2026-09-04 · arXiv (back catalog) · Study
The study investigates students' interest and expectations in a Big Data Engineering course integrated with a Master curricula, as well as ethical implications of using Big Data. An anonymous online survey was conducted with 42 of the 67 students enrolled in the Big Data course offered to Computer Science and...
Why this label for Students' Perception of Big Data Engineering in Higher Education Curricula: Expe
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 survey ceiling: 42 students reported interests and expectations; no learning outcome was measured.
- Grader B: R5 design ceiling: a one-time anonymous survey of 42 master's students with thematic analysis is descriptive (Level 1); R7 perception-only cap also applies; the conference proceedings DOI and ISBN indicate it is peer-reviewed
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Perception or attitudePreprintPerception onlyStudy type: Cross-sectional
2026-08-25 · arXiv (back catalog) · Study (preprint)
The purpose of this study was to analyze patterns of artificial intelligence (AI) use and attitudes toward AI among students, faculty, and administrative staff at a large university specializing in teacher education. The analytical sample comprised 1809 students, 250 faculty members, and 62 administrative staff...
Why this label for The AI Adaptation Gap in Higher Education: Students, Faculty, and Administrative
- Level 1: Descriptive or exploratory: Large attitude and use survey (N=2,121); perception-only descriptive evidence.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: the cross-sectional survey reports controlled associations; R7 caps perception and engagement outcomes at Level 2.
- Grader B: R5 design ceiling: a cross-sectional self-report survey (N=2121); the OLS associations could match Level 2, but a survey is Level 1 and R5 takes the lower; R7 perception-only cap applies anyway
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: K-12 studentsOutcome: Perception or attitudePreprintSmall or imprecisePerception onlyStudy type: Case study / case series
2026-08-19 · arXiv (back catalog) · Study (preprint)
K-12 students often possess creative engineering ideas but lack technical skills to build them, while undergraduates have coding expertise but few opportunities to lead real-world projects or mentor others. The rapid development of AI-assisted tools offers a potential bridge to connect these groups, yet the structure...
Why this label for Multi-Tier Mentorship with AI-Assisted Development: Authentic Engineering for K-
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: the nine-person pilot reports surveys and thematic findings without a comparison group.
- Grader B: R5 design ceiling: a pilot with 4 high-school students and 3 undergraduates, using surveys and thematic analysis; descriptive, and R7 perception-only applies
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Perception or attitudePeer-reviewed (journal reference listed)Perception onlyStudy type: Cross-sectional
2026-07-22 · arXiv (back catalog) · Study
The rise of generative AI (GenAI) in higher education has prompted urgent debates surrounding academic integrity and ethical use. This study examines cross-cultural differences in student perceptions of GenAI use, comparing responses from students at Canadian and South Korean universities.
Why this label for Did Alice Do Wrong? Cross-Cultural Differences in Student Perceptions of Generat
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: the scenario survey compares student judgments but reports no controlled intervention or statistical controls.
- Grader B: R5 design ceiling: cross-sectional scenario-based survey comparing Canadian and Korean students' perceptions, so descriptive Level 1 with a perception-only outcome
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: TeachersOutcome: Perception or attitudePreprintSmall or impreciseStudy type: Case study / case series
2026-07-17 · arXiv (back catalog) · Study (preprint)
While generative AI tools are directly changing how undergraduate computer science is learned and taught, they are also reshaping the relationships between instructors and students. In contrast to existing tool-oriented research on how instructors view and adopt AI, this study investigates how instructors think about...
Why this label for To Police or to Guide: How Higher Education Computer Science Instructors Design
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: 13 instructor interviews are qualitative evidence.
- Grader B: R5 design ceiling: qualitative study of 13 semi-structured instructor interviews, which is descriptive (Level 1).
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Not reportedOutcome: Teacher practicePreprintStudy type: Cross-sectional
2026-07-14 · arXiv (back catalog) · Study (preprint)
With the increased use of generative AI (GenAI) applications such as ChatGPT, higher education institutions (HEIs) have released a range of guidelines and policies to direct adoption within their institutions. In computer science (CS) courses GenAI adoption is especially high and the implications for student learning...
Why this label for A Comparative Analysis of Institutional and Course Generative AI Policies within
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: secondary analysis of policy and syllabus documents is descriptive.
- Grader B: R5 design ceiling: descriptive secondary document analysis of institutional and course AI policies, with no learner outcomes.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: BehaviorPreprintPerception only
2026-06-23 · arXiv (back catalog) · Study (preprint)
The rise of open and distance education has made it more important than ever to have communication tools that are simple, flexible, and good for helping students work together, talk to each other, and feel connected. Researchers have already looked at how instant messaging apps like WhatsApp and Telegram can be used...
Why this label for A Qualitative Comparative Study of Communication in Higher Distance Education
- Level 1: Descriptive or exploratory: Change to 1. Qualitative comparative case study of distance-education students; descriptive real-learner evidence.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R3 genre gate: comparative discussion extends a prior case study without reporting a new systematic study.
- Grader B: R5 design: a qualitative case study and comparative discussion of Viber use at a distance university, which is descriptive (Level 1)
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Perception or attitudePreprintPerception onlyStudy type: Cross-sectional
2026-06-18 · arXiv (back catalog) · Study (preprint)
The rapid integration of generative artificial intelligence (AI) has reshaped the landscape of higher education. Students have embraced tools such as ChatGPT with striking speed, while teaching staff and institutions have responded with greater caution.
Why this label for From Novelty to Normalisation: Tracking Changing Perceptions of AI in Higher Edu
- Level 1: Descriptive or exploratory: Consensus level confirmed against the methodology.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: longitudinal survey tracks reported attitudes and use without an intervention comparison.
- Grader B: R5 design: a repeated survey (n=1,665) of perceptions and reported use, which is descriptive (Level 1)
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Level 1: Descriptive or exploratoryProvisional (rules only)
2026-02-17 · ERIC (peer reviewed) · Study
Question-asking is central to human communication, cognitive development, and learning. While prior research has examined questioning in controlled, non-longitudinal settings, little is known about how the originality and complexity of the questions people ask change over time or how these abilities respond to...
Why this label for Knowledge Reshapes Inquiry by Changing Question Asking Ability and Impacting Aca
- Level 1: Descriptive or exploratory: provisional rules-only grade, pending grader consensus and review: R5 design: descriptive (F3): Descriptive, qualitative, pilot or deployment study with real participants.
- Assessment and grading: matched 'exam'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Level 1: Descriptive or exploratoryLearners: Higher-ed studentsOutcome: Learning or achievementPreprintSmall or impreciseShort exposureResearcher-made measureStudy type: Case study / case series
2025-07-08 · arXiv (back catalog) · Study (preprint)
Retrieval practice is a well-established pedagogical technique known to significantly enhance student learning and knowledge retention. However, generating high-quality retrieval practice questions is often time-consuming and labor intensive for instructors, especially in rapidly evolving technical subjects.
Why this label for Enhancing Student Learning with LLM-Generated Retrieval Practice Questions: An E
- Level 1: Descriptive or exploratory: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 places the week-to-week comparison without equivalence data at Level 2; R6 imprecision lowers it for about 60 students with no interval or power information.
- Grader B: R5: a week with LLM-generated retrieval questions vs a week without (~60 students, no equivalence data) is Level 2, then the R6 imprecision downgrade (n<100, no interval) gives 1; one-week exposure and a researcher-made quiz
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Retrieval practice: matched 'Retrieval Practice'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Technical testLearners: Not a learning studyOutcome: Model performanceStudy type: Technical test (no real learners)
2026-10-19 · arXiv new: cs.CY, cs.HC, cs.CL, cs.AI · Study (preprint)
Teachers increasingly configure educational chatbots through natural-language instructions, but how these bots fail in real-world use remains poorly understood. We introduce EduWild, a dataset of 1,726 conversations with 217 teacher-configured chatbots, each paired with the instructions and configuration behind it,...
Why this label for EduWild: How Teacher-Configured Chatbots Fail in Real-World Use and How to Fix T
- Technical test: R4: outcome is chatbot failure codes per conversation (model behavior), not student learning.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Technical testLearners: Not a learning studyOutcome: Model performanceStudy type: Technical test (no real learners)
2026-10-13 · arXiv search: AI + education · Study (preprint)
Large language models (LLMs) promise to automate one of higher education's most resource-intensive activities: assessing student writing. Existing evaluations are typically limited to a single model, a single prompt and a single institution.
Why this label for Machine-human agreement in marking undergraduate assignments is specific to mode
- Technical test: R4: LLM marking of 761 assignments vs human marks; outcome is model agreement.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Technical testLearners: Not a learning studyOutcome: Model performanceStudy type: Technical test (no real learners)
2026-10-07 · arXiv search: AI + education · Study (preprint)
As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines...
Why this label for Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structu
- Technical test: R4: LLM grading vs instructor scores on an existing dataset; outcome is model performance.
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Technical testLearners: Higher-ed studentsOutcome: Model performancePreprintStudy type: Technical test (no real learners)
2026-10-07 · arXiv (back catalog) · Study (preprint)
As Large Language Models (LLMs) are increasingly adopted for automated grading and feedback in higher education, their structured outputs, including multi-dimensional rubric scores, detailed feedback comments, and improvement suggestions, create an appearance of thorough analytic evaluation. This study examines...
Why this label for Beyond Score Accuracy: Examining the Diagnostic Quality of LLM-Generated Structu
- Technical test: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R4 real-learner gate: student responses provide test data, while the outcomes measure LLM grading and feedback quality.
- Grader B: R4 real-learner gate: compares LLM grades and feedback with instructors on existing student responses; the only outcome is model output quality
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
Technical testProvisional (rules only)Study type: Technical test (no real learners)
2026-10-01 · arXiv physics.ed-ph · Study (preprint)
Earth's ionosphere is primarily generated by the Sun's ionization of atoms in the upper atmosphere. This plasma environment is highly dynamic, exhibits strong spatial and temporal, variations and significantly impacts power systems, navigation, and satellite and ground communications.
Why this label for A Student Sounding Rocket Experiment to Probe Electron Density in the Lower Iono
- Technical test: provisional rules-only grade, pending grader consensus and review: R4 real-learner gate (F1/F2): Outputs of models, simulations or audits; any humans act as raters, references or data sources, not learners.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Technical testLearners: Adults not in a learning role (raters)Outcome: Perception or attitudePreprintPerception onlySmall or impreciseStudy type: Technical test (no real learners)
2026-08-01 · arXiv (back catalog) · Study (preprint)
This innovative practice full paper presents CodeStylist, a web application that provides course-standard-aware code style feedback for early undergraduate programming courses. CodeStylist addresses a common instructional gap: students are expected to follow local conventions for naming, formatting, comments,...
Why this label for CodeStylist: Supporting Early Undergraduate Programmers with Course-Aware Code S
- Technical test: Tool paper; formative review by instructional staff, who are not learners.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R4 real-learner gate: instructional staff rated the tool; anticipated student learning was not measured learning.
- Grader B: R5 design ceiling: formative expert review survey of 18 staff from one course (accepted at FIE 2026), so descriptive Level 1 with only perceived-utility outcomes
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Technical testLearners: Simulated or AI-onlyOutcome: Model performancePreprintStudy type: Technical test (no real learners)
2026-07-14 · arXiv (back catalog) · Study (preprint)
The rise of Artificial Intelligence (AI) enables automatic analysis of large amounts of data. Previously time-consuming and labor-intensive tasks can be completed much more efficiently with the use of AI.
Why this label for Analyzing Curricular Pattern Complexity Using AI to Improve On-Time Graduation R
- Technical test: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R4 real-learner gate: evaluates LLM analysis of curricula without reporting student outcomes.
- Grader B: R4 real-learner gate: LLMs analyze curriculum structures and suggest revisions. No learners took part and no graduation outcome was measured.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
Technical testLearners: Higher-ed studentsOutcome: Model performancePeer-reviewed (journal reference listed)Study type: Technical test (no real learners)
2026-06-12 · arXiv (back catalog) · Study
Educational chatbots powered by large language models (LLMs) show promising effects on learning outcomes, yet most systems delegate pedagogical decisions such as content selection and didactic structuring implicitly to the LLM, making tutoring strategies difficult to trace, evaluate, and reproduce. This paper presents...
Why this label for A didactical-driven teacher assistant for a dimensional modeling course
- Technical test: Two independent AI graders gave the same level.
- Review status: Confirmed against the published method (AI-assisted review): both graders agreed
- Grader A: R4 real-learner gate: authentic student questions are test inputs, while the outcome is system detection precision.
- Grader B: R4 real-learner gate: student questions are only test inputs; the only outcome is the system's detection precision
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
- Tutoring and personalization: matched 'tutoring'
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review): both graders agreed Method
CommentaryStudy type: Expert opinion / commentary
2026-10-19 · Frontiers in Education · Study
Female academic leaders remain systemically underrepresented in senior positions across higher education institutions globally, and this disparity is particularly pronounced across Asian contexts. Drawing on the authors’ collective experience as female academic leaders at IMU University a private health professions...
Why this label for Female academic leadership in private higher education: gender dynamics and lead
- Commentary: Perspective article drawing on authors' experience (R3).
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
CommentaryStudy type: Not a study
2026-10-15 · Higher Ed Dive · News
At the National Association for College Admission Counseling conference, discussions ranged from AI in application essays to international recruitment.
Why this label for How admissions officials are navigating the academic year ahead
- Commentary: news, commentary or policy with no study
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Not a study
2026-10-13 · The Hechinger Report · News
Evan Perez could be the face of a new movement. He’s a first-year student attending Babson College on a full scholarship, majoring in business with a concentration in entrepreneurship.
Why this label for COLUMN: As careers, college and the workforce change rapidly, should high school
- Commentary: news, commentary or policy with no study
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Not a study
2026-10-13 · Higher Ed Dive · News
University leaders discussed application essay strategies in the AI era at the National Association for College Admission Counseling's annual conference.
Why this label for 3 admissions experts on why they require - or nixed - supplemental essays
- Commentary: news, commentary or policy with no study
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryProvisional (rules only)
2026-10-08 · arXiv physics.ed-ph · Study (preprint)
Students often perceive scientific experiments as clean, straightforward procedures with predictable outcomes. Students' beliefs about the nature of experimental physics typically remain inconsistent through the initial stages of undergraduate education.
Why this label for Measuring the Speed of Light with a Time-of-Flight Setup: An Authentic Experimen
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate: No empirical learner data described; conceptual, framework, policy or argument paper.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Not a study
2026-10-05 · EdTech Digest · News
Connected data is only the beginning. Institutions also need clear authority to turn AI insights into action.
Why this label for Before Higher Ed Can Scale AI, It Needs the Right Foundation - and a Reason to B
- Commentary: news, commentary or policy with no study
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryProvisional (rules only)
2026-10-05 · arXiv physics.ed-ph · Study (preprint)
In a bilateral electromechanical transducer, energy exchange across the electromechanical interface allows the electrical termination to modify the mechanical response through electromagnetic feedback. The paper develops this idea from a simple magnetic pendulum, describing the electromechanical coupling through the...
Why this label for Damping by design in a bilateral electromechanical transducer
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate (F7): Self-described conceptual, legal, policy or reflective work with no learner data.
- Preprint: not peer reviewed; capped at Level 3
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryProvisional (rules only)
2026-09-25 · Educational Administration Quarterly (SAGE) · Study
This study examines the role of school leadership in shaping multilingual learners’ (MLs) college-going experiences, drawing on the perspectives of MLs and school counselors. It addresses the gap in understanding how leadership systems can ...
Why this label for How School Leadership Matters in Multilingual Learners’ College Access: Cultural
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate: No empirical learner data described; conceptual, framework, policy or argument paper.
- Inclusion and accessibility: matched 'Multilingual Learners'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Not a study
2026-09-18 · eSchool News · News
In 2023, Catherine Hartmann, a professor at the University of Wyoming, gave students in a history of meditation course a deeply personal assignment: try a contemplative practice and write a reflective analysis on the experience.
Why this label for The oral exam is back, and AI may be the reason
- Commentary: news, commentary or policy with no study
- Assessment and grading: matched 'exam'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Expert opinion / commentary
2026-09-09 · eSchool News · Report / framework
A new national framework from AACTE (American Association of Colleges for Teacher Education) aims to help educator preparation programs (EPPs) navigate the rapidly changing role of artificial intelligence in education.
Why this label for AACTE releases national framework on AI in educator preparation
- Commentary: news, commentary or policy with no study
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Expert opinion / commentary
2026-09-03 · arXiv (back catalog) · Study (preprint)
Contributions: A reflection model suitable for the era of Generative Artificial Intelligence (GenAI) is introduced. The proposed model is an integrated model that extracts features from various existing models and also incorporates technological aspects of GenAI.
Why this label for The 5P Reflection Model for Education in the Generative Artificial Intelligence
- Commentary: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R3 genre gate: the paper proposes a reflection model from existing literature without reporting learner outcomes.
- Grader B: R3 genre gate: it is labelled design-based research, but it analyses existing reflection models and proposes a new one; no participants or outcome data
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
CommentaryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Expert opinion / commentary
2026-07-22 · arXiv (back catalog) · Study (preprint)
Generative AI is changing a basic premise of educational assessment: that submitted work can reliably evidence the human capacities a credential claims to certify. The challenge is not simply whether students use AI, but what remains inferable about learning when some cognitive work has been delegated to a system.
Why this label for What Does the Credential Still Certify? Cognitive Stewardship for AI-Mediated Ed
- Commentary: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R3 genre gate: the paper develops an assessment framework and audits university policies rather than studying learners.
- Grader B: R3 genre gate: conceptual framework plus an LLM-coded audit of 30 university policy documents; no learners, so no learner-level evidence
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
CommentaryLearners: Not reportedOutcome: Not reportedPreprintStudy type: Expert opinion / commentary
2026-07-09 · arXiv (back catalog) · Study (preprint)
Teaching computer science at universities is often structured rather classically and theory oriented. The former refers to "transmission"-style lectures accompanied by exercises which are submitted and graded manually, providing delayed feedback (if any).
Why this label for DOMtutor: Automated Autograding for Logic in Computer Science
- Commentary: Both graders agree; rules version missed it. Consensus level confirmed.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R3 genre gate: proposes an autograding approach without reporting an evaluation.
- Grader B: R3 genre gate: proposes using DOMjudge autograders for theory courses, with no evaluation reported.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
CommentaryLearners: Higher-ed studentsOutcome: Not reportedPreprintStudy type: Expert opinion / commentary
2026-07-08 · arXiv (back catalog) · Study (preprint)
We present documentation of a classroom assignment that teaches runtime monitoring through a creative embedded systems build: an interactive Pachinko game. The assignment centers on a dual-core ESP32 workflow in which students write RTLola specifications for monitors, compile these monitors to C, and deploy them...
Why this label for Experiential Learning of Runtime Monitoring Using Pachinko
- Commentary: Assignment documentation with no study.
- Review status: Confirmed against the published method (AI-assisted review)
- Grader A: R5 design ceiling: documents a classroom assignment without reporting a measured learning outcome.
- Grader B: R3 genre gate: documentation of a classroom assignment with no data or evaluation of student outcomes.
- Rule version: METHODOLOGY v0.3, pipeline v1, rules rules-v0.4
Graded blind by two AI graders against our published method. Confirmed against the published method (AI-assisted review) Method
CommentaryStudy type: Not a study
2026-04-22 · eSchool News · News
Some might worry that the introduction of AI tools in the English classroom will simply lead to more cheating and even worse literacy rates, leaving students unprepared for college and careers that demand strong writing and communication skills.
Why this label for 3 ways students can use AI tools to improve their literacy skills
- Commentary: news, commentary or policy with no study
- Reading and writing: matched 'literacy'
- Academic integrity: matched 'cheating'
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryStudy type: Not a study
2026-03-04 · eSchool News · News
AI is transforming the way students discover, evaluate, and choose colleges, according to a national survey of more than 5,000 high school students conducted by education company EAB.
Why this label for Nearly half of high school students now use AI in college search
- Commentary: news summary of others' work; mentions 'survey', graded C until the cited study is itself graded
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
CommentaryProvisional (rules only)
2025-12-17 · ERIC (peer reviewed) · Study
The transient nature of K-12 English Learner (EL) status and services makes it difficult for researchers to follow their educational pipeline. ELs' backgrounds and experiences vary considerably, with differences in grade of entry into U.S.
Why this label for Variation in English Learners' Programmatic Experiences and Postsecondary Trajec
- Commentary: provisional rules-only grade, pending grader consensus and review: R3 genre gate: No empirical learner data described; conceptual, framework, policy or argument paper.
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
No study citedStudy type: Not a study
2026-10-05 · arXiv search: AI + education · Study (preprint)
We present an AI-based web-oriented tool designed to support formative feedback for oral presentation slides in higher education: AISSA. It allows students to upload their slides before presentation and automatically receive rubric-based quantitative scores and qualitative feedback generated by large language models...
Why this label for AISSA Demo: AI-based Student Slides Analysis Tool for Automated Grading and Feed
- No study cited: Tool demo with no evaluation reported (R1).
- Review status: Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified
Graded by one AI grader from the abstract, on Matt's delegation; not yet human-verified. Method
Not yet classifiedReview of many studies
2026-10-09 · Computers & Education (Elsevier) · Study
Publisher listing (abstract not yet available): Journal article in Computers & Education by Maria Ranieri, Gabriele Biagini. Available online 9 October 2026. Abstract on the publisher page.
Why this label for Artificial Intelligence in Higher Education (2019 - 2026): A Systematic Review o
- Not yet classified: publisher listing with no abstract yet
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
Not yet classified
Listed for the February 2027 issue · Learning and Instruction (Elsevier) · Study
Publisher listing (abstract not yet available): Journal article in Learning and Instruction, Volume 107 by Siyu Jiang, Christoph Stadtfeld. Issue: February 2027. Abstract on the publisher page.
Why this label for The social embeddedness of learning: Social network formation and academic achie
- Not yet classified: publisher listing with no abstract yet
Provisional label from fixed rules applied to the collected text; research items are re-graded by our review pipeline. Method
