Research · Explainer

What internal and external validity say about trusting a result

Internal validity concerns whether a study's comparison is credible, while external validity concerns whether its findings apply beyond the participants and setting investigated. A comparison can be internally credible even when recruitment was restricted or the intervention was delivered under unusual conditions. Recruiting a wider group does not eliminate possible systematic bias.

Validity depends on distinct design features. Allocation, outcome measurement, missing observations and selective reporting bear on the study comparison, while the recruited population and conditions of delivery affect whether its findings can be extended beyond those originally examined. The methodological literature examines these questions separately.

Laboratory seedlings under uniform lighting beside outdoor plants exposed to different conditions
Controlled seedlings in a laboratory beside plants growing under varied outdoor conditions Editorial illustration

What internal validity means in research

Internal validity addresses the credibility of an inference within the investigation that produced it. The relevant question concerns an estimated difference between groups, not a general impression that a study was carefully performed. Cochrane's discussion by Boutron and colleagues (2024) describes bias as systematic deviation from the truth. The direction can vary.

A study's size does not settle the matter. According to Boutron and colleagues (2024), a large trial can produce a precise estimate with a narrow confidence interval even when the estimate remains at high risk of bias. Precision and validity therefore require separate assessments, since more observations mainly change sampling uncertainty rather than correcting a distorted comparison.

Causal inference and competing explanations

Causal inference requires a credible comparison. In their 2024 Cochrane chapter on non-randomised intervention studies, Reeves and colleagues explain that confounding can arise when differences between intervention groups affect the measured outcome, so an observed result cannot safely be assigned to the intervention alone. Study design features matter more than labels such as cohort or case-control.

Random allocation changes that starting position. Higgins and colleagues (2024) explain that successful randomisation keeps known and unknown prognostic factors from determining assignment, so comparison groups have similar expected prognosis on average. That protection addresses the creation of treatment groups. Stuart and colleagues (2011) distinguish that protection from the separate question of who was enrolled.

Internal validity and measurement validity

A comparison also depends on how its outcome is observed. Cochrane separates bias in measuring an outcome from bias arising when participants or clinicians change their behaviour because they know the assigned treatment (Higgins et al., 2024). Those pathways can operate independently. An outcome measured differently between groups may undermine the resulting contrast even if the original assignment was random.

The distinction is visible in research on self-reported recreation participation, where respondents' accounts were examined against recorded activity. Outcome measurement and causal attribution answer related but different methodological questions. Higgins and colleagues (2024) treat participant-reported outcomes as potentially sensitive to knowledge of intervention assignment, particularly when the measure involves judgement.

How internal and external validity differ

The two forms of validity concern different boundaries. Internal validity asks whether the reported contrast is distorted within the analysed group; external validity asks what population or situation the result can speak for. In his 2005 review, Rothwell described external validity as the applicability of randomised-trial findings to a definable set of patients in a particular clinical setting.

Neither property is interchangeable with the other. Boutron and colleagues (2024) explain that a narrowly selected participant group may yield an unbiased estimate for that group despite poor representativeness. Stuart and colleagues (2011) made a related separation between treatment selection and sample selection in their framework for transporting trial findings.

Validity within a study

A randomised comparison concerns outcomes under particular assigned interventions among enrolled participants. Its internal validity depends on the allocation process, what actually happened during follow-up and how the outcome was measured. Higgins and colleagues (2024) assess the risk of bias for a particular result, because the same trial may contain results with different vulnerabilities.

Even a well-protected comparison has a specified meaning. The effect of assignment remains distinct from the effect of adhering to an intervention, as Higgins and colleagues (2024) explain when defining the intention-to-treat and per-protocol targets. Participants who do not follow their assignment complicate the second target. The first target remains relevant to the consequences of assignment itself.

Generalisability beyond the study

Recruitment and assignment address different questions. Stuart and colleagues (2011) compared the measured characteristics of trial participants with those of an identified target population, because successful random assignment among enrolled participants cannot establish that these people resemble everyone a policy might affect. Unmeasured differences remain beyond those diagnostics.

External validity also reaches beyond demographics. Jung and colleagues (2024) reached consensus on a definition covering changes in populations, settings, interventions, outcomes and contextual factors. The reference point must therefore be named. Generalisation to another hospital, school or treatment arrangement may present distinct problems even where participant characteristics appear similar.

Threats to internal validity in research

Threats to internal validity are mechanisms through which a comparison becomes systematically distorted. Boutron and colleagues (2024) distinguish this bias from random error, which makes estimates fluctuate across repeated samples. A large dataset can reduce that fluctuation without cancelling errors linked to study design, measurement or the results that researchers choose to report.

The risk is not always visible in a single number. Cochrane notes that biases can exaggerate or attenuate effects, and their direction may differ between investigations (Boutron et al., 2024). Risk-of-bias assessment therefore evaluates plausible mechanisms rather than claiming to recover the unknown true effect of a particular study.

Confounding occurs when differences between groups besides the intervention are connected with the outcome. In non-randomised work, the exposure may result from routine treatment decisions or individual choices rather than an allocation mechanism. Reeves and colleagues (2024) explain that adjustment for measured confounders may leave residual confounding, and that design features affect the seriousness of the problem.

Some comparisons are also shaped by time. A before-and-after change can coexist with other developments affecting the participants, so the mere sequence of events cannot isolate the intervention. Cochrane's catalogue of non-randomised study features explicitly distinguishes changes over time from differences between comparison groups (Reeves et al., 2024).

Selection matters in a different way when discussing reach. Eligibility rules and recruitment pathways can exclude substantial parts of the eventual target population without necessarily distorting the comparison among those assigned within a trial. The 2006 analysis by Rothwell and the Cochrane distinction described by Boutron and colleagues (2024) place that problem primarily under external validity.

Missing data and outcome measurement

Missing data can distort comparisons. Higgins and colleagues (2024) examine whether missingness is related to the true outcome and whether the pattern differs between randomised groups, since the same number of observed participants in each group does not establish that outcomes are missing independently of their values. Balanced attrition counts do not settle that relationship.

Measurement bias has another route. A participant reporting pain may know which intervention was assigned, and that knowledge could influence the report. Higgins and colleagues (2024) distinguish such judgement-sensitive outcomes from measurements whose recording is unlikely to depend on the observer's awareness. The risk depends on the outcome and the assessment procedure.

Results can disappear after measurement. Boutron and colleagues (2024) describe selective publication as a source of bias in reviews when statistically non-significant or unfavourable studies are harder to locate. A synthesis of only the published reports may therefore represent a selected set of findings, even if every included trial was randomised.

How study design protects internal validity

Protection against bias starts before the first outcome is recorded. Random allocation restricts systematic differences between intervention groups at assignment, but the later conduct of a trial can reintroduce vulnerabilities. Higgins and colleagues (2024) identify separate domains for randomisation, deviations from intended interventions, missing outcomes, measurement and reporting choices.

A strong design specifies which effect is being estimated. For a comparison based on assignment, the treatment decision at baseline defines the intervention group even when some people subsequently deviate from their assigned course. For an effect based on adherence, the analysis must address why adherence differed. Those are distinct questions in the Cochrane framework (Higgins et al., 2024).

Randomisation, allocation concealment and blinding

Randomisation relies on an unpredictable allocation sequence. Concealment concerns whether the next assignment could be known before someone enters a trial; an ostensibly random sequence may lose its protection if investigators can anticipate it. Higgins and colleagues (2024) place both features within the same randomisation-process domain and also examine unexplained baseline differences as possible signs of failure.

Blinding concerns information available after allocation. Participants, intervention providers and outcome assessors need not have the same access to knowledge of assignment. According to Higgins and colleagues (2024), failure to blind participants may permit departures from the intended intervention, whereas an unblinded assessor can introduce error into a judgement-sensitive outcome. One precaution cannot automatically substitute for the other.

A small human study of garlic honey tea and blood pressure illustrates a related reporting problem. Details about randomisation, blinding and group sizes were not reported, making it difficult to assess potential bias. The small sample also limits how confidently the findings can be applied to other populations.

Empirical findings reinforce the distinction. Boutron and colleagues (2024) report that the influence of concealed allocation and blinding varies by the kind of outcome being assessed. Their review of meta-epidemiological evidence identifies larger distortions for subjective outcomes than for objectively measured outcomes. This is an average pattern across collections of studies, not a measured correction for an individual trial.

Comparison groups and prespecified outcomes

The meaning of the comparison depends on the analysis. Higgins and colleagues (2024) distinguish assignment effects from adherence effects because they answer different research questions. Removing participants from an analysis after allocation can affect the intended contrast, particularly when exclusions are connected with outcomes. Missing outcomes need a separate assessment.

Prespecification restricts selective outcome reporting. Higgins and colleagues (2024) identify bias when an outcome analysis is chosen from several available estimates because of its direction, magnitude or statistical significance, even if the report describes its allocation process in considerable detail. The assessment concerns the reported result and its chosen analysis.

Study labels do not resolve those details. Reeves and colleagues (2024) argue that non-randomised investigations should be assessed through their actual features, including prospective decisions and methods to address confounding. In the Seven Countries Study, population-level associations illustrate the distinction between an observed pattern and an experimentally isolated cause.

What affects external validity and generalisability

External validity depends on the relationship between the investigated circumstances and the proposed destination for a result. Patient characteristics are one part of it. In 2006, Rothwell described how healthcare organisation and the timing of care can change the meaning of the same intervention between trial settings. A wider trial population does not automatically settle every transfer question.

External validity needs a defined destination. Stuart and colleagues (2011) used measured characteristics of trial participants and target-population members to assess how closely the enrolled sample resembled the group to which an effect estimate might later be applied. Unrecorded characteristics cannot be compared by that method.

Eligibility criteria and recruited participants

Van Spall and colleagues (2007) systematically sampled 283 randomised trials from 4827 articles published between 1994 and 2006 in selected high-impact general medical journals. They recorded the grounds for excluding participants and assessed the justification reported for each restriction. The sampling frame concerned published medical trials; it was not a census of all experiments or all research fields.

Common medical conditions served as exclusion criteria in 81.3% of those trials, while age was grounds for exclusion in 72.1%, according to Van Spall and colleagues (2007). These percentages describe features of the sampled trial reports. They do not measure how much any one treatment effect differed between excluded and enrolled patients.

Recruitment begins before formal eligibility is applied. The 2006 account by Rothwell describes how the choice of participating clinics and clinicians can already narrow the group available to a trial. In his clinical examples, differences in the speed of diagnosis and treatment across healthcare systems modified the setting in which treatment effects were measured. Eligibility tables alone could not represent that pathway.

Study settings and transportability

Stuart's 2011 example involved elementary schools. The underlying randomised evaluation assigned 37 public elementary schools from five Maryland districts to a prevention programme or control group, and statewide information supplied the target population against which the trial schools could be compared. Those records made the proposed target population explicit.

The statistical proposal used propensity scores to describe the probability of entering a trial given measured characteristics. Stuart and colleagues (2011) discussed matching, weighting and related diagnostics to compare the enrolled sample with its target. Their framework depends on having the relevant effect modifiers observed. Missing information about the factors that alter an effect remains a limit.

Context can also alter how an intervention works. In 2020, Burchett and colleagues argued that superficial resemblance between populations or places provides an incomplete account of generalisability. They focused instead on mechanisms that produce the effect and on conditions that modify those mechanisms. A transfer claim is correspondingly narrower when its mechanism is known only in the original setting.

The problem also surfaced in the earlier methodological exercise published in 2018 by Burchett and colleagues. They tested 11 published applicability tools against a transfer question involving a Swedish weight-management intervention and an English setting. None was judged ideal. The difficulties included unavailable contextual information and incomplete treatment of how the intervention worked.

How internal validity is assessed in published studies

Appraisal requires a named result and a suitable framework. A risk-of-bias judgement for an intervention effect does not automatically describe the whole article, and the same trial may receive different judgements for different outcomes. Cochrane's randomised-trial tool is organised around results rather than a single score for general study quality (Higgins et al., 2024).

Generalisability has fewer agreed measurement conventions. Jung and colleagues (2024) reported that a widely accepted assessment tool had not yet been established; their Delphi process produced a consensus definition and assessment criteria instead. Agreement about the relevant dimensions does not by itself validate a numerical rating for a new population.

Risk of bias in randomised trials

The Cochrane RoB 2 framework, described by Higgins and colleagues (2024), examines the randomisation process, deviations from assigned interventions, missing outcome data, outcome measurement and selective selection of reported results. Each domain contains questions linked to a particular result. Assessors can reach judgements of low risk, some concerns or high risk, supported by reasons rather than a pooled checklist score.

An assessment can distinguish two results measured in the same participants. For instance, participant-reported pain is potentially sensitive to the participant's awareness of assignment, whereas an outcome requiring little observer judgement may be less vulnerable through that particular route. Higgins and colleagues (2024) make this distinction within the measurement domain. Other domains still require independent scrutiny.

Assessment of non-randomised studies

Non-randomised studies require appraisal of how the intervention groups came to differ. Reeves and colleagues (2024) emphasise potential confounding, differences across study designs and the possibility that measured adjustment leaves bias from factors the investigators did not capture. The ROBINS-I framework grades domains of bias from low through moderate and serious to critical. The labels refer to bias risk, rather than the size of an observed effect.

Such studies remain informative for questions not addressed through random allocation. The Cochrane chapter by Reeves and colleagues (2024) includes intervention harms and situations in which routine choices determine the treatment group. Their interpretation remains conditional on the methods used to construct a meaningful comparison and on the outcome being evaluated.

Validity assessments at a glance

Source

Evidence examined

Assessment or reported observation

What it can establish

Higgins et al., 2024

Randomised-trial results

RoB 2 domains and result-specific risk judgements

Risk of internal bias for the assessed result

Reeves et al., 2024

Non-randomised intervention comparisons

Study features, confounding and ROBINS-I domains

Risks within a particular non-randomised comparison

Van Spall et al., 2007

283 sampled trial reports

Exclusions linked to conditions in 81.3% and age in 72.1%

How recruitment criteria restrict the sampled trial populations

Burchett et al., 2018

11 applicability tools

No tool judged ideal in a cross-setting assessment

Limits of existing applicability checklists

Jung et al., 2024

Delphi consensus study

Consensus on 14 external-validity criteria

A proposed assessment basis, not a validated universal score

The comparison table describes different kinds of evidence. Higgins and Reeves assess routes by which a study result may be biased. The work by Van Spall, Burchett and Jung instead concerns conditions for applying or assessing findings outside the original sample. Treating these outputs as one interchangeable score would erase the question each method was designed to answer.

What is proven about internal and external validity and what is not

Internal bias has identifiable mechanisms. Higgins and colleagues (2024) distinguish problems in random allocation, deviations from intended interventions, missing outcomes, measurement and reporting, allowing separate judgements for the components that bear on a particular result rather than one undifferentiated rating for an entire trial. Boutron and colleagues (2024) caution that such judgements cannot reveal the exact size of bias affecting an individual study.

An internally credible result can still have limited reach. Its application elsewhere depends on the participants, intervention and conditions to which the claim is extended.

Generalisability begins with an explicit destination. The 2011 work by Stuart and colleagues sets out ways to compare a study sample with a target population, but its methods depend on measured characteristics that capture relevant effect modifiers. Unobserved differences can survive weighting or matching. This condition remains necessary when the original trial itself is well randomised.

A second limit concerns contextual mechanisms. In 2020, Burchett and colleagues described how population and setting resemblance can miss the processes responsible for an intervention's outcome. Their argument does not establish that every mechanism transfers to a new location; it identifies a question that simple demographic comparisons leave unresolved. Differences in service delivery can be central to that question, as the 2006 examples of Rothwell indicate.

The tools themselves are evolving. In their 2024 consensus study, Jung and colleagues proposed criteria for external validity but also identified the absence of a widely accepted measurement tool. An assessment built around those criteria needs testing before its score could serve as a reliable universal threshold. Separate judgements about bias and applicability therefore remain necessary.

Two further uncertainties require separate evidence. A large sample cannot establish freedom from systematic bias, as Boutron and colleagues (2024) explain, and an unbiased estimate for trial participants cannot automatically specify an effect for a different population. Stuart and colleagues (2011) identify the unresolved transfer question as whether measured characteristics capture the factors that make an intervention effect differ between the trial participants and the proposed target population.

Frequently asked questions

Can a study have high internal validity but low external validity?

Yes. A randomised comparison may protect the estimated effect within a narrow group while leaving its relevance to other participants uncertain. Boutron and colleagues (2024) explicitly distinguish restricted generalisability from bias in the studied population. Rothwell also identifies eligibility criteria, clinical setting and possible changes in an intervention's effect as factors that determine where the result may be applied.

Is random assignment the same as random sampling?

No. Random assignment determines which intervention enrolled participants receive; random sampling concerns who enters the research population. Higgins and colleagues (2024) describe assignment as protection against prognostic imbalance. Stuart and colleagues (2011) focus on the separate task of comparing participants with a target population, using characteristics that were measured before treatment.

Does a large sample size guarantee valid results?

No. A large sample can make an effect estimate precise without removing systematic bias. Boutron and colleagues (2024) distinguish random error from bias and explain that a narrow confidence interval can surround a biased result. Higgins and colleagues (2024) assess allocation, incomplete outcomes, measurement and reporting decisions independently of sample size.

Is ecological validity the same as external validity?

Ecological validity concerns how closely a study's circumstances resemble the environments in which behaviour or interventions occur. External validity covers a question about application across populations, settings, interventions and outcomes. Jung and colleagues (2024) included all those dimensions in their consensus definition. Similarity of setting alone does not establish transferability, as the 2020 analysis by Burchett and colleagues argues.

Can observational studies support causal conclusions?

Observational studies can inform causal conclusions, with limitations. Reeves and colleagues (2024) emphasise study design features and ways of addressing confounding rather than study labels alone. Statistical adjustment accounts for measured characteristics, but differences between groups may remain where relevant factors were not observed.

Sources

  1. Rothwell PM. External validity of randomised controlled trials: to whom do the results of this trial apply? The Lancet. 2005;365(9453):82–93. DOI 10.1016/S0140-6736(04)17670-8.
  2. Higgins JPT, Savović J, Page MJ, Elbers RG, Sterne JAC. Chapter 8: Assessing risk of bias in a randomized trial. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane, 2024. Chapter last updated October 2019. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-08
  3. Reeves BC, Deeks JJ, Higgins JP, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane, 2024. Chapter last updated October 2019. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-24
  4. Rothwell PM. Factors that can affect the external validity of randomised controlled trials. PLOS Clinical Trials. 2006;1(1):e9. DOI 10.1371/journal.pctr.0010009.
  5. Van Spall HGC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297(11):1233–1240. DOI 10.1001/jama.297.11.1233.
  6. Stuart EA, Cole SR, Bradshaw CP, Leaf PJ. The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society Series A. 2011;174(2):369–386. DOI 10.1111/j.1467-985X.2010.00673.x.
  7. Jung A, Braun T, Armijo-Olivo S, Challoumas D, Luedtke K. Consensus on the definition and assessment of external validity of randomized controlled trials: a Delphi study. Research Synthesis Methods. 2024;15(2):288–302. DOI 10.1002/jrsm.1688.
  8. Burchett HED, Blanchard L, Kneale D, Thomas J. Assessing the applicability of public health intervention evaluations from one setting to another: a methodological study of the usability and usefulness of assessment tools and frameworks. Health Research Policy and Systems. 2018;16:88. DOI 10.1186/s12961-018-0364-3.
  9. Burchett HED, Kneale D, Blanchard L, Thomas J. When assessing generalisability, focusing on differences in population or setting alone is insufficient. Trials. 2020;21:286. DOI 10.1186/s13063-020-4178-6.
  10. Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A. Chapter 7: Considering bias and conflicts of interest among the included studies. In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane, 2024. Chapter last updated August 2022. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-07
Eleanor Hayes

Eleanor Hayes — author

Eleanor Hayes writes with an editorial focus on research methodology and study design, including experimental and observational studies, controls, sampling, measurement and the methodological choices that shape scientific conclusions. At Cabdirect, Eleanor cov...

Related