Installing impart

install.packages("devtools")

• Use devtools::install_github to install impart:

devtools::install_github(repo = "jbetz-jhu/impart")

library(impart)

# Available offline in package
vignette("impart_study_design", package = "impart")
vignette("impart_monitoring", package = "impart")
vignette("impart_analyses", package = "impart")

Single Stage Fixed \(n\) Design

• Single Stage: Final analysis - No interim analyses of efficacy/futility
• Fixed \(n\): Recruit fixed number of participants; Last participant last visit (LPLV)
• Minimum Difference: \(\delta = \mu_{1} - \mu_{0}\)
  • Standardized Effect Size: \(d = (\mu_{1} - \mu_{0})/\sigma = \delta/\sigma\)
• \(\sigma\) not of direct interest: nuisance parameter
  • Missing data, Precision Gain: Covariate Adjustment
• Sample Size (\(n\)), Type I Error (\(\alpha\)), Type II Error (\(\beta\))
• Once 3 are known, the other is fixed:
  • Sample Size: \((n \, \vert \, \delta/\sigma, \alpha, 1 - \beta)\)
  • Power: \((1 - \beta \, \vert \, \delta/\sigma, n, \alpha)\)
  • Effect Size: \((\delta/\sigma \, \vert \, n, 1 - \beta, \alpha)\)
  • Type I Error: \((\alpha \, \vert \, \delta/\sigma, n, 1 - \beta)\)

Single Stage Fixed \(n\) Design

# Universal Study Design Parameters
minimum_difference <- 5 # Effect Size: Difference in Means of 5 or greater
alpha <- 0.05 # Type I Error Rate
power <- 0.9 # Statistical Power
test_sides <- 2 # Direction of Alternatives

# Compute Power: Fixed Sample Size
n_7.5 <-
  pwr::pwr.t.test(
    n = NULL,
    d = minimum_difference/7.5, # SD = 7.5
    sig.level = alpha,
    power = power
  )
n_7.5
## 
##      Two-sample t test power calculation 
## 
##               n = 48.26427
##               d = 0.6666667
##       sig.level = 0.05
##           power = 0.9
##     alternative = two.sided
## 
## NOTE: n is number in *each* group

• If \(\sigma = 7.5\): N = 98 provides 90% Power - Conservative to avoid underpowered trial
  • Overestimate? Overpowered; Time, Money, Unnecessary exposure
• Can’t anticipate efficiency gain from adjustment: Additional power

Information Monitored Design:

• Adapt sample size based on the accruing data (Mehta and Tsiatis 2001)
• Identify smallest meaningful difference under \(H_{A}\): \(\delta = \mu_{1} - \mu_{0}\)
• Set a maximum sample size \(n_{max}\): Conservative estimate
• Set a target information level:
  • \(\mathcal{I} = \left(\frac{Z_{\alpha/s} + Z_{\beta}}{\delta}\right)^2\) : \(s\)-sided test with Type I Error \(\alpha\), Power \((1 - \beta)\)
  • \((Z_{\alpha/s} + Z_{\beta})^2: \alpha = 0.05, \beta = 0.10 \approx 10.5; \quad \alpha = 0.05, \beta = 0.20 \approx 7.85;\)
• Monitor the Study:
  • Enroll participants: Collect baseline & outcome data - Analyze periodically
  • Monitor information level: \(\mathcal{I} \approx 1/{\left(SE(\hat{\delta})\right)^2} = 1/Var(\hat{\delta})\)
• Close enrollment: \(n_{max}\) participants or reach target when collection is complete
• Conduct final analysis after last participant’s last visit (LPLV)

Single Stage Information Level:

• required_information_single_stage: Compute target information level
  • delta: Estimand under alternative hypothesis of interest

# Determine information required to achieve desired power at fixed error rate
information_single_stage <-
  impart::required_information_single_stage(
    delta = minimum_difference,
    alpha = alpha,
    sides = 2, 
    power = power
  )

information_single_stage
## [1] 0.4202969

• Conduct analysis when Information = 0.42, Variance = 2.379, SE = 1.542
• Information is abstract, Sample size is concrete; Translation to \(n\) requires:
  • Estimand (Continuous vs. Binary, Difference vs Ratio); Nuisance Parameters

Translate Information to Sample Size

• information_to_n_continuous_1_to_1: Continuous Difference in Means

# Assume Equal Variances: 7.5
approximate_n_sd_7.5 <-
  impart::information_to_n_continuous_1_to_1(
    information = information_single_stage,
    sigma_0 = 7.5,
    sigma_1 = 7.5,
    round_up = TRUE
  )
approximate_n_sd_7.5
##   n_per_arm n_total
## 1        48      96

pwr::pwr.t.test(
  d = minimum_difference/7.5,
  sig.level = alpha,
  power = power
)
## 
##      Two-sample t test power calculation 
## 
##               n = 48.26427
##               d = 0.6666667
##       sig.level = 0.05
##           power = 0.9
##     alternative = two.sided
## 
## NOTE: n is number in *each* group

# Assume Equal Variances: 10
approximate_n_sd_10 <-
  impart::information_to_n_continuous_1_to_1(
    information = information_single_stage,
    sigma_0 = 10,
    sigma_1 = 10,
    round_up = TRUE
  )
approximate_n_sd_10
##   n_per_arm n_total
## 1        85     170

pwr::pwr.t.test(
  d = minimum_difference/10,
  sig.level = alpha,
  power = power
)
## 
##      Two-sample t test power calculation 
## 
##               n = 85.03128
##               d = 0.5
##       sig.level = 0.05
##           power = 0.9
##     alternative = two.sided
## 
## NOTE: n is number in *each* group

Including Covariate Information

• Relative Efficiency: \(RE_{A/U} = Var(\theta_{U})/Var(\theta_{A}) = \mathcal{I}_{A}/\mathcal{I}_{U}\)
  • Solve - adjusted analysis reaches unadjusted threshold: \(\mathcal{I}_{U} = \mathcal{I}_{A}/RE_{A/U}\)

# 20% Relative Efficiency Increase from Covariates
relative_efficiency <- c(1, 1.10, 1.20)
# Information From Final Outcomes + Covariates
impart::information_to_n_continuous_1_to_1(
  information = information_single_stage/relative_efficiency,
  sigma_0 = 10,
  sigma_1 = 10,
  round_up = TRUE
)
##   information sigma_0 sigma_1 n_per_arm n_total
## 1   0.4202969      10      10        85     170
## 2   0.3820881      10      10        77     154
## 3   0.3502474      10      10        71     142

• What if the treatment is more effective/harmful than anticipated?
  • Multi-Stage Designs: Pre-planned Interim Analyses - Stop early for efficacy or futility

Multi-Stage Designs

• Pre-planned interim analyses: Efficacy or Futility
  • Group Sequential Design (GSD): Fractions of final sample size (e.g. 50%, 75%, 100%)
  • Information Monitored Design: Fractions of information target
• Multiple analyses = Multiplicity: \(\alpha\)-spending (Jennison and Turnbull 1999)
• Calculations done using rpact Package in R: Review of GSDs; Web Book on GSDs
• rpact::getDesignGroupSequential(): Set up design with Spending Functions

information_rates <- c(0.50, 0.75, 1.00) # Analyses at 50%, 75%, and 100% of the Total Information
type_of_design <- "asOF" # O'Brien-Fleming Alpha Spending: "asP", "asKD" or "asHSD"
type_beta_spending <- "bsOF" # O'Brien-Fleming Beta Spending: "none", "bsP", "bsKD" or "bsHSD"

trial_design <-
  rpact::getDesignGroupSequential(
    alpha = alpha,
    beta = 1 - power,
    sided = 2,
    informationRates = information_rates,
    typeOfDesign = type_of_design,
    typeBetaSpending = type_beta_spending,
    bindingFutility = FALSE
  )

Adjust Information Level: Multiplicity

• required_information_sequential: Adjust target for multiplicity

# Inflate information level to account for multiple testing
information_multi_stage <-
  impart::required_information_sequential(
    information_single_stage = information_single_stage,
    trial_design = trial_design
  )
information_multi_stage
## [1] 0.4558286

# Inflation Factor
information_multi_stage/information_single_stage
## [1] 1.084539

# Target Information Levels for Analyses
information_multi_stage*information_rates
## [1] 0.2279143 0.3418714 0.4558286

• Inflation factor: fixed sample size design to group sequential design
• Interim Analyses when Information = 0.228, 0.342
• Final analysis when Information = 0.456

Anticipating Analyses

• Logistics of analysis not trivial: Checks, Lock, Export
  • Be able to anticipate when analyses likely to occur
• Planning Interim Monitored Covariate Adjusted Randomized Designs (PICARD)
  • Use on ShinyApps.io - Limited free use
  • Download and run locally from GitHub - Unlimited
• GUI Interface: running same functions as impart
  • Continuous, Binary, Ordinal: Covariate Adjustment, Single/Multi-Stage
• Asymptotic approximations: point estimate, based on nuisance parameters
  • Interval estimates: Simulation - Added to impart in the future

Check Your Understanding:

• Install the pwr package: install.packages("pwr")
• Probability of Event: Treatment = 0.2, Control = 0.1
• Information: 80% Power, \(\alpha\) = 0.05, Two-Sided Test
  • impart::required_information_single_stage
• Sample Size: pwr::pwr.2p.test using effect size pwr::ES.h()
• Compare with impart::information_to_n_binary_1_to_1
• How much information: 3-stage design: 40%, 80%, 100%
  • Pocock Efficacy Boundaries, No Futility
• Sample size for analyses: RE = 1, 1.1, and 1.2?
• Answer posted to Slack: Try first for yourself

Set up Study:

• Create a monitored_design object: initialize_monitored_design()
• Encapsulate entire study design
  • Avoid having to pass many arguments to every function

monitored_design <-
  initialize_monitored_design(
    trial_design = trial_design,
    null_value = 0, # Null Value of Estimand:
    maximum_sample_size = 280,
    information_target = information_multi_stage,
    orthogonalize = TRUE,
    rng_seed_analysis = 54321
  )

Example Dataset: Monitoring

• example_1 - Continuous Outcome: 30, 60, 90, and 120 days post randomization

head(example_1)
##   .id        x_1        x_2        x_3        x_4 tx       y_1       y_2
## 1   1  2.0742970  0.1971432 -0.8425884  0.2794844  0        NA        NA
## 2   2  0.2165473 -0.7384296  0.1315016 -1.2419134  0 -2.376967 -2.575398
## 3   3  0.8294726  0.4997821  1.6932555 -0.4063889  1  2.094655 13.615512
## 4   4 -1.0206893 -0.2189937 -1.7719120  0.1936013  1 -6.433176  3.015335
## 5   5 -0.0417332  0.9282685  0.8078133  0.9317145  1  5.653307  8.839702
## 6   6  0.7275778  1.1756811  0.0226265 -0.2556343  0        NA        NA
##         y_3       y_4 .enrollment_time     .t_1      .t_2     .t_3     .t_4
## 1        NA        NA         33.28136 77.28136 107.28136 137.2814 167.2814
## 2        NA        NA         33.69242 74.02729  96.49186 137.6924 167.6924
## 3 20.402979 24.403011         37.69694 73.32954 102.44161 129.2825 152.6411
## 4  5.473575 -6.548695         39.40100 78.95480  95.29012 123.9621 156.4938
## 5 21.906824  4.005459         41.32190 71.43593 102.43171 141.4073 150.4507
## 6        NA        NA         55.24537 99.24537 129.24537 159.2454 189.2454

• “Wide” format: one individual per row, indicated by .id
• Baseline Covariates: x_1, x_2, …, x_4; Treatment assignment tx
• Randomization Time: .enrollment_time
• Outcomes y_1, y_2, y_3 and y_4 assessed at times .t_1, .t_2, .t_3 and .t_4
• 276 observations in data

Monitoring Ongoing Studies

• Outcome \(Y\) is missing: Missed visit/assessment? Not yet observed?
• Notation:
  • Baseline Covariates: \(X_{1}, X_{2}, \ldots, X_{p}\)
  • Treatment Assignment: \(A\)
  • Post-Randomization Outcomes: \(Y_{1}, Y_{2}, \ldots, Y_{J}\)
  • Time - Study Initiation to Randomization: \(T_{A}\)
  • Time - Study Initiation to Outcome Assessment: \(T_{Y_{1}}, T_{Y_{2}}, \ldots, T_{Y_{J}}\)
  • Study Window \(j\) Opens/Closes: \(w^{o}_{j}, \, w^{c}_{j}\)
• Timing of data accrual important: How quickly information is accruing
  • Not needed for computing information at a single point in time

Missing vs. Not-Yet-Observed

• Notation for data at a particular point after study initiation:
  • \(t\): Days since study initiation
  • \(t - T_{A}\): Time on study for participant \(i\)
• Assessment has occurred: \(T^{(t)}_{Y_{ij}}\) is time of assessment
• Study Window Not Yet Closed \((t - T_{A_{i}} \le w^{c}_{j})\): Treat as missing \(T^{(t)}_{Y_{ij}} = NA\)
  • Outcome could still potentially be observed per protocol: Not yet observed
• Study Window Closed \((t - T_{A_{i}} > w^{c}_{j})\): Set time to \(T^{(t)}_{Y_{ij}} = T_{A_{i}} + w^{c}_{j}\)
  • Known Missing: Last possible time the outcome could be assessed per protocol
• Outcome Indicator: \(R^{(t)}_{ij} = 1\) if \(Y_{ij}\) is observed by time \(t\)
  • \(R^{(t)}_{ij} = 0\) if \(Y_{ij}\) is missing and \((t - T_{A_{i}} > w^{c}_{j})\)
  • \(R^{(t)}_{ij} = NA\) if \(Y_{ij}\) is missing and \((t - T_{A_{i}} \le w^{c}_{j})\)

Preparing Data for Analyses

• Check data for consistency; Subset to relevant data
• What role does each variable play? Order, Timing of Outcomes
• What data was known at any given point in the study

prepared_final_data <-
  prepare_monitored_study_data(
    data = example_1,
    study_time = 2075, # Days: Study Initiation to Data Lock
    id_variable = ".id",
    covariates_variables = c("x_1", "x_2", "x_3", "x_4"),
    enrollment_time_variable = ".enrollment_time",
    treatment_variable = "tx",
    outcome_variables = c("y_1", "y_2", "y_3", "y_4"),
    outcome_time_variables = c(".t_1", ".t_2", ".t_3", ".t_4"), 
    # Observe missingness 1 week after target study visit
    observe_missing_times = c(30, 60, 90, 120) + 7
  )

names(prepared_final_data)
## [1] "data"          "original_data" "variables"     "study_time"

• Checked dataset, original data: metadata - role of variables, study time
• Used by other impart functions with a prepared_data argument

Monitoring Completion

# Plot events two years into the study
plot_outcome_counts(
  prepared_data = prepared_final_data,
  study_time = 731, # 2 Years
  type = "ct" # "t" Total, "c" Complete
)

Monitoring Completion

# Plot events two years into the study
plot_outcome_counts(
  prepared_data = prepared_final_data,
  study_time = 731, # 2 Years
  type = "ct" # "t" Total, "c" Complete
)

# Plot events two years into the study
plot_outcome_counts(
  prepared_data = prepared_final_data,
  study_time = 731, # 2 Years
  type = "c" # "c" Complete
)

Tabulating Event Counts:

• Count events contributing to information: Randomizations, Outcomes
• count_outcomes: Returns a data.frame with the study time of each event
• Determine when e.g. 60 primary outcomes have been observed

example_1_counts <-
  count_outcomes(
    prepared_data = prepared_final_data
  )

head(example_1_counts)
##           event     time count_total count_complete
## 1 randomization 33.28136           1              1
## 2 randomization 33.69242           2              2
## 3 randomization 37.69694           3              3
## 4 randomization 39.40100           4              4
## 5 randomization 41.32190           5              5
## 6 randomization 55.24537           6              6

Tabulating Event Counts:

# Timing of first n observations of `y_4`
subset(
  example_1_counts,
  event == "y_4"
) |> head()
##      event     time count_total count_complete
## 516    y_4 150.4507           1              1
## 316    y_4 152.6411           2              2
## 416    y_4 156.4938           3              3
## 1106   y_4 167.2814           4             NA
## 284    y_4 167.6924           5             NA
## 716    y_4 171.0864           6              4

# Find when n observations of `y_4` are first available:
subset(
  example_1_counts,
  event == "y_4" & count_complete == 60
)
##     event     time count_total count_complete
## 817   y_4 740.0624          81             60

Computing Trajectory of Information

• Want to monitor information acccrual over time: \(\hat{\mathcal{I}}(t)\)
  • Revert a dataset to data available at time \(t_{1}, t_{2}, \ldots t_{T} < t\)
  • Compute \(\hat{\mathcal{I}}(t_{1}), \hat{\mathcal{I}}(t_{2}), \ldots, \hat{\mathcal{I}}(t_{T})\)
  • Trajectory of information accrual: Model & Smoothed
• Could keep track of multiple different datasets…
• Instead: recover from a single dataset: for each \(t^{*} \in \{t_{1}, t_{2}, \ldots t_{T}\}\)
  • Keep participants randomized by study time \(t^{*}\): \(T_{A_{i}} < t^{*}\)
  • Compute \(R^{(t^{*})}_{ij}\) earlier: Observed/Missing/Not-Yet-Observed at time \(t^{*}\)
• data_at_time_t: Revert a prepared dataset to time \(t^{*}\)

Using data_at_time_t()

• Look at example_1 at 90 days after study initiation

data_90 <-
  data_at_time_t(
    prepared_data = prepared_final_data,
    study_time = 90
  )

head(data_90$data[, 1:10], n = 6)
##   .id        x_1        x_2        x_3        x_4 tx       .e .r_1     .t_1       y_1
## 1   1  2.0742970  0.1971432 -0.8425884  0.2794844  0 33.28136    0 77.28136        NA
## 2   2  0.2165473 -0.7384296  0.1315016 -1.2419134  0 33.69242    1 74.02729 -2.376967
## 3   3  0.8294726  0.4997821  1.6932555 -0.4063889  1 37.69694    1 73.32954  2.094655
## 4   4 -1.0206893 -0.2189937 -1.7719120  0.1936013  1 39.40100    1 78.95480 -6.433176
## 5   5 -0.0417332  0.9282685  0.8078133  0.9317145  1 41.32190    1 71.43593  5.653307
## 6   6  0.7275778  1.1756811  0.0226265 -0.2556343  0 55.24537   NA       NA        NA

• Participant 1 missed their 1st assessment: .r_1 is 0
• Participants 2-5 have had their 1st assessment: .r_1 is 1
• Participant 6 missed 1st assessment - Not known by day 90: .r_1 is NA

Questions so far?

• Notation for a time-varying dataset
• How to track events: Randomization & Outcomes
• How to find times of particular events
• How to revert data to an earlier study time

Computing Information:

• Revert the final data back: 60 primary outcomes observed
• What was the information level at this point in study?

data_n_final_60 <-
  data_at_time_t(
    prepared_data = prepared_final_data,
    study_time = 
      # Time when 60 final outcomes are observed:
      ceiling(
        subset(
          example_1_counts,
          event == "y_4" & count_complete == 60
        )$time
      )
  )

# Extract data reverted to 60 complete outcomes
data_n_60 <- data_n_final_60$data

Computing Information: Single

• Specify estimation_function: Any function that returns "estimate"
• Specify estimation_arguments: List of any arguments for estimation_function
  • Formulas, Family of Outcome, etc.

information_n_60 <-
  estimate_information(
    data = data_n_60,
    monitored_design = NULL, # Pass previous analyses in multi-stage designs
    estimation_function = standardization,
    estimation_arguments =
      list(
        estimand = "difference",
        y0_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        y1_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        family = gaussian,
        treatment_column = "tx",
        outcome_indicator_column = ".r_4"
      ),
    orthogonalize = TRUE,
    return_results = TRUE, # FALSE: return only covariance & information
    rng_seed = 23456
  )

Computing Information: Single

• Extract information when 60 primary outcomes observed:

information_n_60$covariance_orthogonal
##           estimates
## estimates  5.382429
information_n_60$information_orthogonal
## estimates 
## 0.1857897

• What about computing an information trajectory?

Compute Information Trajectory

data_n_60_trajectory <- 
  information_trajectory(
    prepared_data = data_n_final_60,
    monitored_design = NULL,
    estimation_function = standardization,
    estimation_arguments =
      list(
        estimand = "difference",
        y0_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        y1_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        family = gaussian,
        treatment_column = "tx",
        outcome_indicator_column = ".r_4"
      ),
    orthogonalize = TRUE,
    n_min = 30, # Start at `n_min` primary outcomes
    n_increment = 2, # Increment forward
    rng_seed = 23456
  )

head(data_n_60_trajectory)
##      times randomization y_1 y_2 y_3 y_4 information information_lag_1 information_change information_pct_change
## 1 453.1467            56  44  41  36  30  0.04122672                NA                 NA                     NA
## 2 466.1324            58  47  42  37  32  0.03642663        0.04122672       -0.004800091              -13.17742
## 3 468.1530            58  47  42  37  34  0.05526950        0.03642663        0.018842867               34.09270
## 4 479.5912            59  48  42  38  36  0.07829199        0.05526950        0.023022490               29.40593
## 5 511.2584            64  52  47  41  38  0.08848096        0.07829199        0.010188968               11.51544
## 6 514.7433            64  52  47  41  40  0.09940316        0.08848096        0.010922199               10.98778

Smoothing the Trajectory

• Plot trajectory: Robust fit: Regression calibration (deming::theilsen())

Curtailing Enrollment:

• At final analysis: no not-yet-observed outcomes:
• Final Sample Size: \(n(t)\mathcal{I}/\hat{\mathcal{I}}_{c}(t)\)
  • \(n(t)\): Participants with complete outcomes: count_outcomes
  • \(\mathcal{I}\): Information target - stored in monitored_design
  • \(\hat{\mathcal{I}}_{c}(t)\): Estimated information: completed last visit
    • Calculate using estimate_information

Questions so far?

• Setting up information monitored design: initialize_monitored_design()
• Computing information with: estimate_information()
• Computing trajectory of information: information_trajectory()

Performing Pre-Specified Analyses

• Syntax for analyses very similar to computing information:

interim_analysis_1 <-
  monitored_analysis(
    data = interim_1_data,
    monitored_design = monitored_design,
    estimation_function = standardization,
    estimation_arguments = 
      list(
        estimand = "difference",
        y0_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        y1_formula = y_4 ~ x_1 + x_2 + x_3 + x_4,
        family = gaussian,
        treatment_column = "tx",
        outcome_indicator_column = ".r_4"
      )
    )

Performing Pre-Specified Analyses

• Result of monitored_analysis is a monitored_design object
  • Original Design + Each analysis nested within
  • Estimates, Covariance, Information, Bootstrap information
  • Test Statistics, Decisions, Details
  • sessionInfo(): environment, packages, versions
• Multi-Stage: Pass previous analysis as monitored_design argument
• Any method used - wrapper function: returns "estimate"
  • Bootstrap: long or wide data; Preserve covariance across analyses

Future Directions

• Convenience Functions: estimating final sample size
• More Estimators: AIPW, TMLE
• Variance Correction: Small sample size adjustments: preserve \(\alpha\)
• Worked Examples: implementing new methods, estimands in impart
• Time-To-Event Outcomes: Survival Probability, RMST

Wrapping Up

• impart covers planning, monitoring, and analyses
• Planning: simple add-on to existing workflow
• Monitoring: Tabulating/Plotting data accrual & information
• Vignettes contain worked examples: expanded (other outcome types, methods)
• Check GitHub for updates & issues
  • Your feedback & experience extremely helpful