A new kind of trial: How an old machine learning algorithm can help researchers run faster and cheaper trials
Do we really need a new kind of trial?
Discovering new knowledge in the medical field is no easy feat. Statistical robustness is not negotiable, so researchers take a host of precautions to ensure that the data they painstakingly collect are capable of producing a meaningful conclusion.
This is typically done with a randomised controlled trial (RCT). If you want to understand causal links in a complicated system — from the effects of a new medicine on human disease to the effect of a certain policy on human behaviour — an RCT is usually the best tool in your toolkit.
But what if you’re not necessarily interested in understanding all of the causal connections? What if, for example, you’re looking at a policy question? Or instead of trying to understand how one particular strategy works (or doesn’t work) in detail, what if you simply want to know which among a set of strategies performs the best?
Although an RCT can answer these questions, it can come with a hefty price tag. This means that policy researchers who want to make data-driven decisions, but who don’t have the resources for a full-scale RCT, can find themselves in a sticky position.
With this in mind, Deakin’s Applied Artificial Intelligence Institute (A²I²) and Global Obesity Centre (GLOBE) have teamed up to develop a new approach to clinical trials that don’t fit neatly into the RCT box.
To be precise, we developed a method for contexts where:
- You have a number of different strategies (or interventions) to consider
- You want to find the best strategy
- You are not interested in the ordering of strategies that are not the best (in other words, you just want to separate “the best” from “the rest”)
How did the first trial work?
Our first trial of this kind investigated how to increase the number of conversations that patients have with their GP about physical activity. The aim was not to understand the detailed psychology of GP-patient interactions, but instead to find the best strategy among 8 while working with a very limited research budget.
GLOBE recruited 26 GPs at 13 different clinics for a 12 week trial. We started by collecting 2 weeks of baseline data for each GP before any strategies were implemented. Following the establishment of a baseline, we wanted to maximise the proportion of consultations during which physical activity is discussed. To this end, we used a so-called “Multi-Armed Bandit algorithm” (MAB) to schedule which strategy should be tried each week. Our algorithm starts by assuming that all strategies have no effect, but then gradually uses the evidence collected from trials to update its belief.
After 7 weeks of using the algorithm in this way, 3 strategies showed consistently strong results. With only 3 weeks of the trial left, the remaining weeks were focused on the 3 most promising strategies to increase our confidence about which one works best.
Benefits of this approach
The MAB algorithm we used is very efficient at steadily zeroing in on the best option. Like any other method, it needs to have a reasonable sample size to produce meaningful results, but once given a certain number of samples to play with, it can use them very efficiently.
Other benefits apply to areas where people are directly affected by the outcome of each sample of the trial. For instance, if you were trialling strategies for addressing a mental health problem, then the MAB approach benefits trial participants because it slowly tends toward the more promising strategy over time, thereby meaning that the effect of participating in the trial is beneficial for both the researcher and the patient.
Future work
To our knowledge, this is the first time that this kind of trial has been used (nice work GLOBE!). However, there are many other settings where the MAB algorithm could make clinical trials faster and cheaper. These new applications will require adjustments based on the specific needs of the research community, and we look forward to making a general-purpose tool to accelerate health-care research!
Read the paper at: https://www.nature.com/articles/s41746-019-0205-y
Written by Stephan Jacobs