Method
How items are collected, how research is graded, and what the evidence levels mean.
What we grade
We grade a specific finding about learners, not a journal, product or article as a whole. We grade how much the source lets you trust the claim, not whether the result was positive: a well-run study with a null result can earn a high level.
Standards we borrow from
Our rules borrow concepts from the What Works Clearinghouse standards (study design, attrition, baseline equivalence), the ESSA tiers of evidence, Evidence for ESSA (caution about researcher-made measures and very short studies) and GRADE (downgrading for risk of bias, indirectness and imprecision). We do not issue ratings under any of them, and our levels are not official tiers.
Evidence levels
Levels go from 1 to 4. Higher is stronger. Level 4 is the strongest evidence we give, and only a person can award it after reading the whole study. A level tells you how much to trust a study's result, not whether the result was good: a strong study can find that a tool did not help. Letters (T, C, N, ?) mark items that are not studies of real learners, or that we cannot grade yet.
Evidence levels describe how much a source lets you trust a claim about learners, based on study design, who took part and what was measured. They are not endorsements, and they are not official WWC, ESSA or GRADE ratings. See all labels and badges, with examples.
Students were put into groups by chance (a randomized trial), or the study compared students using something with similar students who were not, and showed both groups started out about the same. Preprints, company-linked and very short studies can reach Level 3 at most.
Two things go together, even after accounting for other factors, or groups were compared without showing they started out alike. Cannot show cause on its own. Studies that only measure feelings or engagement can reach Level 2 at most.
Describes what happened with real students or teachers, usually without a comparison group. Useful for ideas, not for telling whether something works.
Tested the AI, used only pretend (computer-simulated) students, or had people only rate the AI's answers. Tells us about the tool, not about student learning.
Opinion, policy, ideas, a review without a careful, stated search, or a news story about someone else's work. Not itself a study.
A product launch or claim with no study we could find behind it. Treat the claim as unproven.
Not enough information to grade yet, or our rules disagree and a review is pending.
Study types
Every item also gets a study type, using the names from the standard education evidence pyramid. The type describes the design; the evidence level still decides how much to trust a finding about learners. Strongest designs sit at the top.
| Study type (strongest to weakest) | How it maps to our levels |
|---|---|
| 1. Systematic review / meta-analysis | Synthesizes many studies. Our method rates a review no higher than the studies it includes, and Level 4 needs a person's review. For now, the automatic pass gives a meta-analysis a temporary Level 3 and a review without combined results Level 1, until a person checks. Narrative reviews are Commentary (C). |
| 2. Randomized controlled trial | Level 3 (Level 4 only after human review). |
| 3. Quasi-experimental | Level 3 (Level 4 only after human review). |
| 4. Cohort / case-control | Level 2. |
| 5. Cross-sectional | Level 2 or Level 1. |
| 6. Case study / case series | Level 1. |
| 7. Expert opinion / commentary | Commentary (C). |
Outside the pyramid. Technical tests (T) run only on AI models, simulated students or benchmarks, with no real learners, so they sit outside the pyramid however rigorous they are. News and launches with no study are tagged Not a study.
Hierarchy adapted from common education evidence frameworks (WWC, ESSA, EEF). Study types are assigned by fixed rules first and by an AI reviewer only when the design is ambiguous; they never change an evidence level.
Key rules
- Real learners first. Studies run only on AI models, simulated students or benchmarks are always a Technical test, however rigorous.
- Caps. Preprints can reach Level 3 at most until peer reviewed. Studies that only measure feelings or engagement can reach Level 2 at most. Studies of under a week (or a single session) can reach Level 3 at most. Company-linked studies are held at Level 3 until a person checks funding and independence; that hold lifts if the checks pass.
- Reviews of many studies. Our method says a review should be rated no higher than the studies it includes, and Level 4 needs a person to check how the review searched for, judged and combined the studies. Our automatic first pass does not check the studies inside yet. When it has enough information and the review is about learners, it gives a meta-analysis a temporary Level 3 and a systematic or scoping review that does not combine results into one number a temporary Level 1. Treat these as unconfirmed: a 3 may be too high, and a 1 means we have not graded the studies inside, not that they are weak.
- Downgrades. A level drops for a mismatched population, fewer than 100 learners (or fewer than 2 classes or schools per condition) with no confidence interval or power information (our own threshold, not a WWC number), clustered designs analyzed as individuals, or researcher-made measures with no reliability evidence.
- Level 4 is never automatic. A strong design with real learners and learning outcomes can only be confirmed by a person reading the full paper (attrition, baseline equivalence, measures, preregistration). No item currently holds Level 4.
- Unknown means unknown. Missing details are shown as not reported, never guessed.
How research is graded
- Scripts collect items from public feeds: education news outlets, peer-reviewed journals and preprint servers such as arXiv. For the main tracker, items that do not mention AI and education are filtered out. The Research beyond AI section collects broader education research from ERIC (peer reviewed only), four education journals and arXiv physics education, capped at about 40 items a week. OpenAlex is used only to look up exact publication dates by DOI.
- Research items get a provisional level from fixed, published rules applied to the abstract.
- Two independent AI graders and blindly grade each study against this method. When they disagree, the lower level is used and the study is flagged.
- Every study at Levels 1 to 3, and every flagged study, then gets a second review against this method. For the October 2026 back catalog that review was done by an AI reviewer on the editor's behalf; no label has yet been verified by a human, and each card shows its review status. Off-topic items are dropped.
- News, commentary and launches are labeled by rule: news coverage is labeled Commentary until the study it describes is located and graded, and launches with no traceable study are No study cited. Research items not yet through the grader review show a Provisional badge.
The back catalog started as 283 arXiv studies collected over the past year. 140 are in the tracker; 173 were dropped as off-topic (not about education or learning): 50 off-topic (queue decision), 123 off-topic (keyword screen, not in queue).
Learning principle tags
Retrieval practice, Spacing and interleaving, Feedback, Cognitive load, Worked examples and explicit instruction, Metacognition and self-regulation, Motivation and engagement, Reading and writing, Inclusion and accessibility, Tutoring and personalization. Tags are applied by keyword rules when an item mentions a principle; a tag means the item discusses it, not that it is good evidence about it. Separate topic tags cover assessment and grading, academic integrity, and policy and governance.
Field tags
Every item also gets up to three Field tags from keyword rules: Instruction and curriculum, Cognitive science and learning, Assessment, Teacher effectiveness and efficacy, Teacher preparation and PD, School leadership, Edtech (non-AI), Literacy, Math, Science and STEM, Special education, Early childhood, Higher education, Equity and policy, Other education research. The Edtech (non-AI) tag is used only in Research beyond AI. Field tags never change an evidence level.
Disclaimer
Sponsor policy
Evidence Signal accepts sponsors whose products or services serve educators, but sponsorship never buys coverage, placement in the tracker, or a better evidence level. Every sponsored item is labeled "Sponsored" and kept separate from the automated tracker. Sponsors cannot review or edit content before it is published. We do not accept sponsors that sell student data, make claims of learning results they won't back with a source, or market directly to children. Labels on a sponsor's own products are set by the same fixed rules as everyone else's, and we may decline or end any sponsorship that conflicts with these terms.
Corrections
Spot a wrong label, broken link or error? Email a correction. Fixes are listed in the corrections log.