Senior Research Integrity Editor, Lisa Bero 

As systematic reviewers, we synthesize the best available evidence to inform the decisions of the public, patients, clinicians, and policy makers. But what happens when the evidence is fatally flawed? 

Lisa Bero argues that systematic reviewers are uniquely positioned – and ethically obligated – to detect problematic studies and champion research integrity.

The threat of problematic studies

There are many reasons a study might be considered untrustworthy, ranging from fraud, deliberate data fabrication and scientific misconduct, to poor research practices or honest errors.

Whatever the cause of the issues, the inclusion of a problematic study could influence a review’s outcomes, subsequently affecting clinical decision making and the care people receive. Trials with research integrity problems have been shown to exaggerate the effectiveness of the interventions. In addition, research has shown that excluding studies with identified research integrity issues can alter the results of a review. Estimates on the prevalence of problematic clinical trials are highly varied, but the potential impact on systematic reviews cannot be denied.  

Why systematic reviewers?

Systematic reviewers are uniquely placed for several reasons. As part of the review process, systematic reviewers look across a whole body of evidence. This gives a distinct perspective and the opportunity to pick up hallmarks of problematic studies, such as discrepancies, plausibility and potential errors. In addition, a key step in systematic reviews is to assess the risk of bias of included studies, so systematic reviewers are used to scrutinizing every study they include in a review.

Finally, if systematic reviews exclude problematic studies, this has the potential to decrease their use in wider scientific literature. Research has shown that studies that are rated in systematic reviews as having a high risk of bias subsequently receive fewer citations. If reviews also excluded studies based on trustworthiness, it might help reduce the attention these studies receive.

Identifying problematic studies

The challenge is that identifying problematic studies is not straightforward. In terms of tools to identify problematic studies, it is a bit of a Wild West; there are many methods and a lack of consensus on which method is most effective. Some researchers are reluctant to even disclose their methods in case those producing fraudulent data can then escape them!  

Using unvalidated methods is risky. For example, it could lead to incorrectly labelling studies as problematic. Moreover, some existing methods can introduce inequities by, for example, focusing on an investigator’s country of origin or where the paper was published rather than the study itself.

This is why we need a systematic, validated method to assess research integrity. Cochrane is collaborating to develop such a tool – Investigating Problematic Clinical Trials in Systematic Reviews (INSPECT-SR) – to identify potentially problematic randomized controlled trials in systematic reviews. The tool is informed by a scoping review of existing tools and the views of researchers involved in research integrity. The aim is that, following development and testing, INSPECT-SR will be recommended for identifying problematic studies in Cochrane reviews.

Taking action

If you identify a potentially problematic study in your systematic review, what should you do? Cochrane has a policy for its reviewers. The bottom line is, the study should not be included in the Cochrane review and should not contribute results to the review findings.

Beyond this, reviewers have an obligation to try to remove the study from the literature, and Cochrane provides templates for this. Although as a reviewer you can’t carry out a formal investigation, you can raise your concerns with someone who can. Cochrane's policy recommends that reviewers initially contact study authors for clarification or additional information. If this doesn’t resolve the issue, reviewers are advised to contact the journal(s) where the study was published.  

Our role goes beyond summarizing studies; we must actively identify and exclude untrustworthy research to protect the credibility of the systematic reviews and the decisions they inform. Systematic reviewers should also commit to transparency. We should share methods for identifying problematic studies, and we should share any research integrity investigations to prevent duplication.  

There is only so much systematic reviewers can do. We can draw attention to the issue, but we can’t prevent these studies from being published, or issue retractions. And even with a standardized method such as INPECT-SR, applying trustworthiness checks will add more steps to an already intensive process.  

But here’s why we should champion research integrity. If we can’t trust the primary literature, the trustworthiness of systematic reviews becomes even more important. Systematic reviews are only as credible as the studies they include. And after investing months or years into a review – don't we owe it to ourselves and the people who rely on our work to make sure each included study is real and trustworthy? Ultimately, our obligation is to the public, patients, healthcare professionals, and policymakers who rely on our reviews.

So, embrace tools like INSPECT-SR, share your methods, take the time to check for trustworthiness, and speak up about your concerns. Upholding research integrity is not an optional extra, it is central to our mission. 

Learn more about the INSPECT-SR projectRead the full article