Predictive probability of success

Predictive probability of success (PPOS) is a statistics concept commonly used in pharmaceutical industry including by health authority to support decision making. In clinical trials, PPOS is the probability of observing a success in the future based on existing data.

Types of PPOS

• Classification based on type of end point: Normal, binary, time to event.
• Classification based on the relationship between the trial providing data and the trial to be predicted

1. Cross trial PPOS: using data from one trial to predict the other trial
2. Within trial PPOS: using data at interim analysis to predict the same trial at final analysis

• Classification based on the relationship between the end point(s) with data and the end point to be predicted

1. 1 to 1 PPOS: using one end point to predict the same end point
2. 1 to 1* PPOS: using one end point to predict another different but correlated end point

Relationship with conditional power and predictive power

Conditional power is the probability of observing a statistically significance assuming the parameter equals to a specific value.^[1] This is a Frequentist Statistical power. Conditional power is often criticized for assuming the parameter equals to a specific value which is not known to be true. If the true value of the parameter is known, there is no need to do an experiment. Predictive power addresses this issue assuming the parameter has a specific distribution. Predictive power is a Bayesian power. A parameter in Bayesian setting is a parameter. Predictive power is a function of a parameter(s), therefore predictive power is also a variable. Both conditional power and predictive power use statistical significance as success criteria. However statistical significance is often not enough to define success. For example health authorities often require the magnitude of treatment effect to be bigger than statistical significance to support a registration decision. To address this issue, predictive power can be extended to the concept of PPOS. The success criteria for PPOS is not restricted to statistical significance. It can be something else such as clinical meaningful results. PPOS is conditional probability conditioned on a random variable, therefore it is also a random variable. The observed value is just a realization of the random variable. ^[2]

Relationship with posterior probability of success

Posterior probability of success is calculated from posterior distribution. PPOS is calculated from predictive distribution. Posterior distribution is the summary of uncertainties about the parameter. Predictive distribution has not only the uncertainty about parameter but also the uncertainty about estimating parameter using data. Posterior distribution and predictive distribution have same mean, but former has smaller variance.

Common issues in current practice of PPOS

PPOS is a conditional probability conditioned on randomly observed data and hence is a random variable itself. Currently common practice of PPOS uses only its point estimate in applications. This can be misleading. For a variable, the amount of uncertainty is an important part of the story. To address this issue, Tang^[3] introduced PPOS credible interval to quantify the amount of its uncertainty. Tang advocates to use both PPOS point estimate and credible interval in applications such as decision making and clinical trial designs. Another common issue is the mixed use of posterior probability of success and PPOS. As described in the previous section, the 2 statistics are measured in 2 different metrics, comparing them is like comparing apples and oranges.

Applications in clinical trial design

PPOS can be used to design futility interim for a big confirmatory trials or pilot trials.

Pilot trial design using PPOS

Traditional pilot trial design is typically done by controlling type I error rate and power for detecting a specific parameter value. The goal of a pilot trials such as a phase II trial is usually not to support registration. Therefore it doesn't make sense to control type I error rate especially a big type I error as typically done in a phase II trial. A pilot trail usually provides evidence to support a Go/No Go decision for a confirmatory trial. Therefore it makes more sense to design a trial based on PPOS. To support a No/Go decision, traditional methods require the PPOS to be small. However the PPOS can be small just due to chance. To solve this issue, we can require the PPOS credible interval to be tight such that the PPOS calculation is supported by sufficient information and hence PPOS is not small just due to chance. Finding an optimal design is equivalent to find the solution to the following 2 equations.

1. PPOS=PPOS1
2. upper bound of PPOS credible interval=PPOS2

where PPOS1 and PPOS2 are some user-defined cutoff values. The first equation ensures that the PPOS is small such that not too many trials will be prevented entering next stage to guard against false negative. The first equation also ensures that the PPOS is not too small such that not too many trials will enter the next stage to guard against false positive. The second equation ensures that the PPOS credible interval is tight such that the PPOS calculation is supported by sufficient information. The second equation also ensures that the PPOS credible interval is not too tight such that it won't demand too much resource.

Futility interim design using PPOS

Traditional futility interim is designed based on beta spending. However beta spending doesn't have intuitive interpretation. Therefore it is difficult to communicate with non-statistician colleagues. Since PPOS has intuitive interpretation, it makes more sense to design futility interim using PPOS. To declare futility, we mandate the PPOS to be small and PPOS calculation is supported by sufficient information. Finding the optimal design is equivalent to solving the following 2 equations.

1. PPOS=PPOS1
2. upper bound of PPOS credible interval=PPOS2

References

1. Ankerst, J; Ankerst, D. Handbook of statistics in clinical oncology (2nd ed.). p. 232.
2. Tang, Z. "PPOS design". slideshare.
3. Tang, Z. "Optimal futility interim design: a predictive probability of success approach with time to event end point". Journal of Biopharmaceutical Statistics, 2015, 25(6), 1312-1319. doi:10.1080/10543406.2014.983646.