The ProofPilot Blog - Design, Launch & Participant in Research Studies

When the Scientific Gold Standard Isn’t the Right Tool to Determine What Works


On any given week, you’ll probably read a dozen or so news articles based on studies that show what’s going to cure a disease or maximize longevity. The study results treated with the most credibility are those that use the randomized controlled trial (or RCT for short).

The RCT is the gold standard of medical evidence.

In an RCT, researchers randomize study participants into different treatment groups and compare outcomes over time. The results of an RCT provide the strongest scientifically verifiable proof of a treatment’s effectiveness. And, it’s not just used for healthcare treatments. Marketers, politicians, and social-service providers put the most weight on well-conducted RCT results when making decisions about what works and what doesn’t.

Recently, however, there’s been some chatter that the randomized controlled trial is losing favor. The 21st Century Cures Act signed into law by President Obama allows the FDA to consider real world data when approving pharmaceuticals instead of requiring an RCT. A well-publicized New York Times article on the lack of evidence for flossing made scientists and the public stand up in arms. Everyone knows you should floss.

The RCT isn’t going anywhere. It’s the scientific method applied to humans. But, it is not always the right research technique to use.

Sometimes expert opinion, secondary data analysis or observational studies are better to provide the right answers. Here are the key reasons why an RCT might not be the right research tool.

When expert opinion is in overwhelming agreement.

A NYTimes article from August 2016 suggested that there is limited evidence to support the benefits of flossing, but it does not tell the whole story. The researchers found very few randomized controlled trials on flossing. Why?

Because nearly any expert you ask will say flossing does work.

The expert opinion so overwhelming, an RCT seems unnecessary. No ethical review panel would approve a study in which a portion of the study population was asked not to floss for years on end. It puts those participants health at unnecessary risk for absolutely no benefit.


What should be done instead of an RCT?: In this particular case, researchers asked the wrong question.

Everyone knows that flossing works. But, many of us still don’t do it — or don’t do it often enough.

Instead of asking what is the evidence that flossing works, we should be testing interventions, products, and programs that show promise of increasing flossing frequency and effectiveness..

No alternative treatments exist and assigning people to a no-treatment control would be ethically problematic.

Infectious disease outbreaks like Ebola require quick and immediate action. Ebola is quickly fatal, and there are few treatment options. When an experimental treatment is potentially your only hope for saving lives and stopping an epidemic, you don’t do an RCT. You’re not going to withhold that treatment randomly from some of your patients in what is essentially Russian roulette. You’re going to give it to everyone.


What should be done instead of an RCT?

Researchers can do an observational studies without a control when no alternative treatments exist. Everyone in observational studies gets treatments, and researchers watch the outcomes. The only drawback is there’s no comparison, so it’s harder to screen rule out other variables as having an impact on the results.

You might be asking, isn’t withholding treatment from those in need unethical in all situations?

It’s not typically an all or none situation. When a study is looking at a treatment innovation or improvement, researchers can provide “standard of care” as the control. For slower moving issues, modified randomized controlled trial designs where the control is current best practice, or where the treatment is only delayed within the control arm provide the scientific comparisons without barring treatment from those in need.

When testing a treatment for a very rare problem, and it would be impossible to recruit enough participants for a meaningful analysis.

There are approximately 7,000 rare diseases and disorders, 95% of them have no approved treatment. There are an estimated 260 people in the world that suffer from aquagenic urticaria, otherwise known a severe allergy to water. Those who suffer are confined to their homes. Recruiting an adequate number of participants would be impossible — particularly if it required the individuals to be within travel distance to a medical facility.


What should be done instead of an RCT?

Instead, for these rare diseases outcome studies, or “real world data” are the best options. These studies provide treatments and interventions to all participants and track results.

The randomized controlled trial remains the gold standard and is the design of choice for most human subject research studies when budget, logistics, and ethics are not a concern.

However, it is not the only option. Other techniques or even adjustments to your research question may be more appropriate.

You'll Also Be Interested In

Other Stories in the ProofPilot Blog