For published studies, this command calculates (1) how much bias there must be in an estimate to invalidate/sustain an inference; (2) the impact of an omitted variable necessary to invalidate/sustain an inference for a regression coefficient.

pkonfound(
  est_eff,
  std_err,
  n_obs,
  n_covariates = 1,
  alpha = 0.05,
  tails = 2,
  index = "RIR",
  nu = 0,
  n_treat = NULL,
  switch_trm = TRUE,
  model_type = "ols",
  a = NULL,
  b = NULL,
  c = NULL,
  d = NULL,
  two_by_two_table = NULL,
  test = "fisher",
  replace = "control",
  to_return = "print"
)

Arguments

est_eff

the estimated effect (such as an unstandardized beta coefficient or a group mean difference)

std_err

the standard error of the estimate of the unstandardized regression coefficient

n_obs

the number of observations in the sample

n_covariates

the number of covariates in the regression model

alpha

probability of rejecting the null hypothesis (defaults to 0.05)

tails

integer whether hypothesis testing is one-tailed (1) or two-tailed (2; defaults to 2)

index

whether output is RIR or IT (impact threshold); defaults to "RIR"

nu

what hypothesis to be tested; defaults to testing whether est_eff is significantly different from 0

n_treat

the number of cases associated with the treatment condition; applicable only when model_type = "logistic"

switch_trm

whether to switch the treatment and control cases; defaults to FALSE; applicable only when model_type = "logistic"

model_type

the type of model being estimated; defaults to "ols" for a linear regression model; the other option is "logistic"

a

cell is the number of cases in the control group showing unsuccessful results

b

cell is the number of cases in the control group showing successful results

c

cell is the number of cases in the treatment group showing unsuccessful results

d

cell is the number of cases in the treatment group showing successful results

two_by_two_table

table that is a matrix or can be coerced to one (data.frame, tibble, tribble) from which the a, b, c, and d arguments can be extracted

test

whether using Fisher's Exact Test or A chi-square test; defaults to Fisher's Exact Test

replace

whether using entire sample or the control group to calculate the base rate; default is the entire sample

to_return

whether to return a data.frame (by specifying this argument to equal "raw_output" for use in other analyses) or a plot ("plot"); default is to print ("print") the output to the console; can specify a vector of output to return

Value

prints the bias and the number of cases that would have to be replaced with cases for which there is no effect to invalidate the inference

Examples

# using pkonfound for linear models pkonfound(2, .4, 100, 3)
#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 60.3 % of the estimate would have to be due to bias. #> This is based on a threshold of 0.794 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 60 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 60). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(-2.2, .65, 200, 3)
#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 41.732 % of the estimate would have to be due to bias. #> This is based on a threshold of -1.282 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 83 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 83). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(.5, 3, 200, 3)
#> Robustness of Inference to Replacement (RIR): #> To sustain an inference, 91.549 % of the estimate would have to be due to bias. #> This is based on a threshold of 5.916 for statistical significance (alpha = 0.05). #> #> To sustain an inference, 183 of the cases with 0 effect would have to be replaced with cases at the threshold of inference (RIR = 183). #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(-0.2, 0.103, 20888, 3, n_treat = 17888, model_type = "logistic")
#> Conclusion: #> User-entered Table: #> Fail Success #> Control 2882 118 #> Treatment 17308 580 #> #> #> Note: Values have been rounded to the nearest integer. #> This may cause a little change to the estimated effect for the Implied Table. #> #> To sustain an inference for a negative treatment effect, you would need to replace 1 treatment success cases #> with null hypothesis cases (RIR = 1). #> This is equivalent to transferring 1 case from treatment success to treatment failure, #> as shown, from the Implied Table to the Transfer Table. #> Transfer Table: #> Fail Success #> Control 2882 118 #> Treatment 17309 579 #> #> For the Implied Table, we have an estimate of -0.200, with a SE of 0.103 and a t-ratio of -1.946. #> and a t-ratio of -0.200. #> RIR: #> RIR = 1
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(2, .4, 100, 3, to_return = "thresh_plot")
pkonfound(2, .4, 100, 3, to_return = "corr_plot")
pkonfound_output <- pkonfound(2, .4, 200, 3, to_return = c("raw_output", "thresh_plot", "corr_plot") )
#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 60.557 % of the estimate would have to be due to bias. #> This is based on a threshold of 0.789 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 121 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 121). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460. #>
#> Print output created by default. Created 3 other forms of output. Use list indexing or run summary() on the output to see how to access.
summary(pkonfound_output)
#> Created 3 forms of output. To access type: #> #> pkonfound_output$raw_output #> pkonfound_output$thresh_plot #> pkonfound_output$corr_plot
pkonfound_output$raw_output
#> # A tibble: 1 x 8 #> action inference percent_bias_to… replace_null_ca… unstd_beta beta_threshhold #> <chr> <chr> <dbl> <dbl> <dbl> <dbl> #> 1 to_inv… reject_n… 60.6 121 2 0.789 #> # … with 2 more variables: omitted_variable_corr <dbl>, itcv <dbl>
pkonfound_output$thresh_plot
pkonfound_output$corr_plot
# using pkonfound for a 2x2 table pkonfound(a = 35, b = 17, c = 17, d = 38)
#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 13 treatment success #> cases with null hypothesis cases (RIR = 13). #> This is equivalent to transferring 9 cases from treatment success to treatment failure. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.278, with p-value of 0.051: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 26 29 #> #> RIR: #> RIR = 13
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(a = 35, b = 17, c = 17, d = 38, alpha = 0.01)
#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 9 treatment success #> cases with null hypothesis cases (RIR = 9). #> This is equivalent to transferring 6 cases from treatment success to treatment failure. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.835, with p-value of 0.011: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 23 32 #> #> RIR: #> RIR = 9
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(a = 35, b = 17, c = 17, d = 38, alpha = 0.01, switch_trm = FALSE)
#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 18 control failure #> cases with null hypothesis cases (RIR = 18). #> This is equivalent to transferring 6 cases from control failure to control success. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.790, with p-value of 0.012: #> Transfer Table: #> Fail Success #> Control 29 23 #> Treatment 17 38 #> #> RIR: #> RIR = 18
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().
pkonfound(a = 35, b = 17, c = 17, d = 38, test = "chisq")
#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 15 treatment success #> cases with null hypothesis cases (RIR = 15). #> This is equivalent to transferring 10 cases from treatment success to treatment failure. #> For the User-entered Table, we have a Pearson's chi square of 14.176, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have a Pearson's chi square of 3.640, with p-value of 0.056: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 27 28 #> #> RIR: #> RIR = 15
#> For other forms of output, run ?pkonfound and inspect the to_return argument
#> For models fit in R, consider use of konfound().