The ProofPilot Blog - Design, Launch & Participant in Research Studies

Alternatives to the Randomized Controlled Trial

ProofPilot’s goal is to “democratize” techniques that determine “what works to improve lives.” Here are alternatives to the randomized controlled trial that are better suited to your research question, a better place to start and easier to pull off.

In health and wellness, when we think “proof,” we think randomized controlled trial (RCT). The RCT is the gold standard tool to “determine what works.” It sits at the pinnacle of the evidence triangle. RCTs aren’t limited to pharma studies. They’re used in social-services, behavior change, and even website optimization.

Creating evidence on “what works” is a series of repeated studies. As you repeat, you use increasingly robust sets of research techniques. Each stage provides important building blocks to the next. And, even if you get to the pinnacle, that randomized controlled trials should even be repeated. That’s part of the scientific process. The same results from repeated studies improve the strength of your evidence.

As you think about your research program you don’t need to jump right to the gold standard. Here’s a list of other research techniques that can help you along your path to determining “what works.” This list isn’t one that’ll match up with research methods textbooks. It’s a list of techniques easy to identify, design, and complete yourself.

Lived Experience and the anecdote. We’re constantly experimenting consciously and subconsciously. Likewise, providers have professional lived experience. They learn from every patient or customer they serve. Interesting, unusual and promising cases become stories told at a professional level.

The results of these personal and professional stories are anecdotes. They are stories we tell our friends, family, and coworkers. Professionals often report unique situations via case reports.

Anecdotes are generally not scientifically valid. They aren’t replicated, generally difficult to verify and don’t include hard data.

Yet, Anecdotes are an important part of the evidence development process. They inspire a hypothesis for further testing.

What you can do: Anecdotes are everywhere. Start looking for anecdotes that seem to repeat the same findings. Use them to form a hypothesis.

Expert Opinion and Repeated Anecdote. Professionals who see lots of customer or patients often begin to see trends. These trends become expert opinions. Expert opinion is a formally accepted component of the evidence building process. When many experts start to form a similar hypothesis based on experience, there’s likely something there to take a look at.

ProofPilot also believes there is also real value in spotting trends on social-media and popular press. Patients and consumers technically are experts in themselves.

While still not verifiable with a standardized hard process and outcome data, your able to start sensing a trend. If the trend all points in a series of similar directions, you’ve strengthened your hypothesis.

What you can do: Keep a watch out with Google Keyword searchers. Follow vocal patients and customers on social media. Read journal articles and case reports by noted experts. Run a survey of key customers, patient advocates, and professionals. Use the information to strengthen your hypothesis. These resources can also be useful as you think about what’s useful to measure in a potential formal study.

Retrospective Data Analysis. Many digital health organizations and healthcare providers have access to large amounts of patient and user data. Retrospective studies evaluate existing materials collected in the past.

Retrospective analysis can be great. You don’t have to recruit new participants. So, you can analyze large amounts of data and explore findings.

But, there are some drawbacks. Retrospective studies rely on the quality of record keeping. Available data is limited. What data is available likely wasn’t created for your research question. Therefore, even if you have data, it might not include the right elements to address your hypothesis. And it’s unlikely the data is collected in a systematic format that is required to reduce bias.

What you can do: In the rare situation you’ve got a good bit of data, use it to identify new questions. With a prospective controlled study, you can confirm and expand upon retrospective findings. Existing data can help you identify where there are gaps.

Single Arm Prospective Longitudinal Study. Prospective studies are studies actively creating study experiences into the future. A single arm observational study is one of the simplest studies you can conduct. Everyone in the study has exactly the same experience. There’s no randomization. The only comparisons are changes over time. Tracking change over time can identify trends and be insightful.

There are significant benefits over retrospective data analysis. You are planning what data you want to collect to address your hypothesis. You are carefully controlling the dosage and timing of treatments and interventions. This systematic control makes the results more generalizable and scientifically valid.

What you can do: Maybe your research question has no treatments or interventions. You can run a single arm observational study. There’s no point to randomization. You could be looking at a treatment or intervention where comparative treatments wouldn’t be possible or ethical. You can design and launch a single arm prospective longitudinal study on ProofPilot.

Multi-Arm Trials and Natural Experiments. An arm in a study is a set of participant experiences. These experiences are often, but not always unique treatments and interventions. By differing the experience in each arm, researchers can determine differences.

Randomized controlled trials have two or more arms. Participants are randomly assigned to one of the arm experiences. Participants and professionals often have no idea which arms they are in. RCTs aren’t the only multi-arm trial.

A natural experiment tracking something happening, but you’re not pulling the levers. It’s happening naturally. In that situation, you may assign participants to different comparative arms based on demographic criteria or other specific participant data point.

It doesn’t have to be a natural experiment. Perhaps you want to give participants a choice. Or you want to compare different types of individuals. A longitudinal multi-arm trial is a great option.

What you can do: Start with a single arm study on ProofPilot, add a new arm, and adjust the conditional arm assignment criteria. Upon analysis compare between the different arms.

With these techniques to build evidence, you may decide an RCT isn’t necessary. Or, you could decide to use a non-traditional RCT technique which we’ll discover in a subsequent post. You could generate worthwhile data from these non-randomized techniques that warrant further study with an RCT. Either way, the RCT is not the start and finish of the research process.

You'll Also Be Interested In

Other Stories in the ProofPilot Blog