Sometimes the design, implementation, or analysis of a study can lead to a systematic error: a a deviation from the truth that would have been uncovered by a perfectly designed study. This is known as bias. Risk of bias therefore refers to the risk that a study’s results will overestimate or underestimate a true intervention effect. A study’s risk of bias can vary in magnitude depending on the number and severity of sources of bias that are present.

Risk of bias is often assessed in randomized controlled trials (RCTs), but bias can also trickle down into systematic reviews and meta-analyses — after all, these syntheses are composed of individual studies! Bias from randomized and/or non-randomized studies can make its way into systematic reviews and meta-analyses in two ways. First, if the included studies tend to overestimate or underestimate a true effect, then the average effect will also be off. Several studies with unrealistically high numbers for a result will give an unrealistically high average. Second, studies with “positive” results may be more likely to get published than those that don’t find an effect. Thus, the conclusions drawn by a meta-analysis may paint an unrealistic picture of the truth due to this non-reporting bias.^[1]

Risk of bias should not be confused with imprecision. Bias refers to systematic error, meaning that multiple replications of the same study would, on average, reach the wrong conclusion. Imprecision refers to random error, meaning that multiple replications of the same study would lead to different effect estimates because of sampling variation, even if they would give the right answer on average. Risk of bias should also not be confused with a lack of external validity. External validity refers to the ability of the results of a study to be generalized to other populations. For example, a lot of psychological research has been critiqued for its lack of external validity due to biased sampling from Western, educated, relatively wealthy societies.^[2]

Bias can exist across multiple study designs, including systematic reviews (as discussed above) and observational/non-randomized studies. Here are some of the most common tools used to assess the risk of bias in observational/non-randomized studies:

Risk of Bias in Non-randomised Studies - of Interventions (ROBINS-I): this tool assesses the risk of bias in non-randomized studies that compare the effects of two or more health interventions, including cohort studies, case-control studies, controlled before-and-after studies, and controlled trials in which participants are not randomized (often referred to as “quasi-randomized” studies). Evaluating the risk of bias in a non-randomized study involves assessing domains of bias including confounding, selection of the participants, classification of the interventions, deviations from the intended interventions, missing data, measurement of the outcomes, and selection of the reported result.^[3]

The Newcastle-Ottawa Scale (NOS): the NOS assesses the quality of non-randomized studies (in particular case-control and cohort studies) across three broad domains: the selection of the study groups, the comparability of the study groups, and the ascertainment of the outcome of interest. This scale uses a star system to judge study quality and risk of bias. A maximum of 9 stars is possible, with scores of 7–9 indicating a low risk of bias, 4–6 points indicating a high risk of bias, and 0–3 points indicating a very high risk of bias.^[4]

Two of the most common tools for assessing risk of bias in systematic reviews are the Risk of Bias in Systematic Reviews (ROBIS) tool and AMSTAR 2:

ROBIS: this tool assesses the risk of bias in systematic reviews broadly related to healthcare settings, including reviews of interventions, diagnosis, prognosis, and etiology. The tool is completed in 3 phases that involve: 1) assessing the relevance of the review; 2) identifying concerns with the review process; and 3) judging the risk of bias. The bias domains assessed in phase 2 include the studies’ eligibility criteria; the review’s identification and selection of studies, data collection and study appraisal; and the review’s synthesis and findings. Phase 3 includes an overall assessment of the risk of bias in the interpretation of the review findings.^[5]

AMSTAR 2: this tool evaluates systematic reviews of randomized or non-randomized studies of healthcare interventions and includes a total of 16 domains. Domains of critical importance relate to whether the protocol was registered before the review was commenced, the adequacy of the literature search, the justification for excluding individual studies, the risk of bias being included in the review, the appropriateness of the meta-analytical methods, the consideration of the risk of bias when interpreting the results, and the assessment of the presence and impact of publication bias. This tool provides a rating of its overall confidence in a review, which can be high (no or one non-critical weakness), moderate (more than one non-critical weakness), low (one critical flaw with or without some non-critical weaknesses), or critically low (more than one critical flaw with or without non-critical weaknesses).^[6]

Given the importance of randomized controlled trials in assessing causality, the focus for the remainder of this glossary entry will be on randomized controlled trials (RCTs).

The most widely used tool for assessing the risk of bias in RCTs is the Cochrane Risk of Bias 2 (RoB) tool.^[7][8] This tool assesses the risk of bias in studies using 5 types of bias: bias arising from the randomization process, bias due to deviations from intended interventions, bias due to missing outcome data, bias in measurement of the outcome, and bias in the selection of the reported result.

Within each domain, a series of questions aims to elicit information about features of the trial that are relevant to the risk of bias. A judgment about the risk of bias arising from each domain is assessed based on carefully considered answers from a reviewer to specific signaling questions. These judgments about risk of bias are be classified as ‘low’, ‘high’, or 'some concerns’. The considerations for each of the five domains include the following:

Bias arising from the randomization process, also known as sequence generation and allocation concealment: Proper randomization is essential to avoid the influence of known or unknown factors on a participant's assignment to an intervention group. Improper randomization can lead to the effect estimate of the result being biased by confounding. Random assignment should be based on a process that includes an element of chance, and neither the participants nor the trial personnel should be aware of forthcoming allocations until after recruitment has been confirmed.

Bias due to deviations from intended interventions, also known as performance bias: The intended intervention refers to the intervention specified in the protocol for a trial. Deviation from the intervention may involve administering interventions that are inconsistent with the protocol, a failure to implement the protocol interventions as intended, or a failure of the participants to adhere to their assigned intervention.

Bias due to missing outcome data, also known as attrition bias: Missing measurements of the outcome can bias the effect estimate of the intervention and may arise due to several reasons: participants may withdraw from the study, be lost to follow up, or drop out of the study; participants may fail to attend a study visit where the outcome is measured; participants may attend a study visit but fail to provide relevant outcome data; data or records may be lost or unavailable; or participants may become unable to experience the outcome (e.g., by passing away during the study). Generally, the risk of bias will increase as the amount of missing outcome data increases.

Bias in the measurement of the outcome, also known as detection bias: Errors in measuring an outcome can depend on several factors, including: whether the measurement of the outcome was appropriate; whether the measurement of the outcome differed (or could have differed) between the intervention groups; who the outcome assessor was (e.g., was it the participant if the outcome was self-reported, the intervention provider, or an observer not directly involved in the intervention’s provision); whether the outcome assessor was blinded to the intervention assignments; and whether measurement of the outcome could have been influenced by the knowledge (of the assessor/trial personnel or the participant) of the intervention received.

Bias in the selection of the reported result, also known as reporting bias: Bias can arise when the reported result is selected from multiple intervention effect estimates because of on the chosen result’s direction, magnitude, or statistical significance. This typically arises from a desire for findings that support vested interests and/or the researchers’ preconceptions, or from the researchers’ desire to report results that are noteworthy and merit publication in a high-impact journal. This bias can be minimized if the trial’s results are analyzed according to a pre-specified plan that was finalized before the unblinded outcome data were available; conversely, failure to create and adhere to a pre-specified plan increases the likelihood of reporting bias.

Based on the judgments provided for each domain, the assessors may then judge a trial’s overall risk of bias. A “low risk of bias” indicates that the trial’s primary result is judged to be at a low risk of bias for all of the domains. “Some concerns” indicates that a trial is judged to raise some concerns in at least one domain, but is not at a high risk of bias in any domain. A “high risk of bias” indicates either, that the trial is judged to be at a high risk of bias in at least one domain, or that multiple domains have “some concerns” that lower confidence in the result.

Ultimately, a trial’s risk of bias is an indicator of the confidence we have in a particular result. The lower the risk of bias, the higher the confidence we can have that that result reflects reality. Studies with a high risk of bias should be interpreted cautiously.