Labels
Every label on Evidence Signal, in plain language.
Evidence levels
Levels go from 1 to 4. Higher is stronger. Level 4 is the strongest evidence we give, and only a person can award it after reading the whole study. A level tells you how much to trust a study's result, not whether the result was good: a strong study can find that a tool did not help. Letters (T, C, N, ?) mark items that are not studies of real learners, or that we cannot grade yet.
Evidence levels describe how much a source lets you trust a claim about learners, based on study design, who took part and what was measured. They are not endorsements, and they are not official WWC, ESSA or GRADE ratings.
What it means: A randomized trial with real students, or a review of many controlled studies with real students that searched for studies carefully and checked their quality. Either way, it must be published after expert review, measure learning, report how big the effect was, and pass a person's full check.
Example: A district randomly gives an AI tutor to half of its 40 algebra classes for a school year, and a reviewer confirms few students dropped out and the groups started out alike.
How much to trust it: The most trust we give. Still check that the students and setting look like yours. No item holds this level yet.
What it means: Students were put into groups by chance (a randomized trial), or the study compared students using something with similar students who were not, and showed both groups started out about the same.
Example: A study compares 30 classes using an AI tutor with 30 similar classes that are not, after showing both scored about the same on a fall pretest.
How much to trust it: Good reason to take it seriously, but not proof. A single study that was not randomized cannot go above Level 3. Randomized trials and careful reviews can move to Level 4 only after a person checks the full paper, and a check can still leave a study here. A review of many studies may show a temporary Level 3 (see Reviews of many studies below).
- Why preprints stop here: A preprint is a paper posted online before other experts have reviewed it. Until that check happens, mistakes may not have been caught, so we never rate a preprint above Level 3. Once a reviewed version comes out, we grade it again.
- Why company-linked studies stop here: If a study's authors, funders or publisher have a stake in the product, it can reach Level 3 at most until a person checks who paid for it and whether the team was independent. The hold lifts if those checks pass, but passing does not by itself raise the level.
- Why short studies stop here: If students used the tool for only one session or less than a week, the study can reach Level 3 at most, because one short try rarely shows lasting learning.
What it means: The study found that two things go together, even after accounting for other factors, or it compared groups but did not show the groups started out alike. On its own, this kind of study cannot show cause. Some stronger studies also land here because of a cap or a drop (see below).
Example: Students who used a chatbot more had higher grades, but they may have been stronger students to begin with.
How much to trust it: Worth noticing, not enough to change practice on its own. Studies that only measure how people felt about a tool, or how engaged they were, can reach Level 2 at most, because liking a tool is not the same as learning more.
What it means: The study usually describes what happened with real students or teachers, without a comparison group. A stronger study can also end up here if serious problems lower its level.
Example: Interviews with 12 teachers about using AI for lesson planning, or a one-class pilot with a test before and after.
How much to trust it: Useful for ideas and for spotting problems. It cannot tell you whether something works.
What it means: The study tested the AI, used only pretend (computer-simulated) students, or had people only rate the AI's answers. It does not test whether real students learned from the tool.
Example: Checking how often a chatbot gives correct answers to math questions.
How much to trust it: Tells you about the tool, nothing about whether students learn from it.
What it means: Opinion, policy, ideas, a review that summarizes research without a careful, stated search, or a news story about someone else's work.
Example: A policy brief on AI guidelines, or a news article about a new study we have not graded yet.
How much to trust it: Can help you think, but it is not itself a study. For news, the study it describes can be graded on its own once it is located.
What it means: A product launch or claim with no study we could find behind it.
Example: A company says its app "boosts learning by 30%" with no research linked.
How much to trust it: Treat the claim as unproven.
What it means: We do not have enough information about the study to grade it yet, or our rules disagree and a review is pending.
Example: A journal listing that shows only a title, with no summary.
How much to trust it: Unknown for now. We grade it once details are available.
A study drops one level for each serious problem, but never below Level 1. Problems include: fewer than 100 learners with no range showing how precise the result is and no power check; fewer than 2 classes or schools per group; a test made by the researchers or the product's maker with no proof it is reliable; whole classes assigned, but results analyzed as if each student had been assigned separately; or a mismatch between the study and the claim (for example, adults standing in for K-12 students). Full details are in the method's key rules.
Reviews of many studies
Meta-analyses and systematic reviews combine results from many studies. They can be the strongest evidence there is, but only if the studies inside are strong.
Our method says a review should be rated no higher than the studies it includes, and Level 4 needs a person to check how the review searched for, judged and combined the studies.
Our automatic first pass does not check the studies inside yet. When it has enough information and the review is about learners, it gives a meta-analysis a temporary Level 3 and a systematic or scoping review that does not combine results into one number a temporary Level 1. Treat these as unconfirmed: a 3 may be too high, and a 1 means we have not graded the studies inside, not that they are weak.
A review that summarizes studies without a careful, stated search (a narrative review) is Commentary (C). Our first pass usually marks a review of work with no real learners as Commentary too.
Look for the Review of many studies badge. It shows how many studies were included when the item's summary says so.
Badges
- Preprint: Not yet peer reviewed. Caps the level at 3.
- Peer-reviewed (journal reference listed): The arXiv record lists a journal reference or DOI for a published version.
- Small or imprecise: Fewer than 100 learners and no confidence interval or power analysis.
- Short exposure: A single session or less than a week of use.
- Researcher-made measure: The outcome test was built by the study team with no reliability evidence.
- Vendor-linked: An author, funder or publisher has a stake in the product.
- Perception only: Measures how people feel about a tool, not whether they learned more.
- Open data or materials: Data or materials are publicly available.
- Report (grey literature): A report that did not go through journal peer review.
- Provisional (rules only): Graded by our fixed rules only; the two-grader review has not run yet.
- Simulated learners: Uses AI agents or synthetic students instead of real learners.
- Preregistered: The plan was registered before data were collected.
- Effect size reported: Reports a standardized effect, ideally with a confidence interval.
- Level-4 candidate: Could qualify for the top level, held at Level 3 until a person reads the full paper.
- Review of many studies: A meta-analysis or systematic review. Shows the number of included studies when the summary states it. Its level is provisional until a person checks the studies inside.
Who took part and what was measured
Reviewed study cards show who took part (K-12 students, higher-ed students, adult learners, teachers, adults acting only as raters, simulated or AI-only, or not reported) and the outcome measured (learning or achievement, behavior, behavior or engagement, perception or attitude, teacher practice, model performance, or not reported). Each card also shows its review status: Confirmed against the published method (AI-assisted review), or human-verified.