For published studies, this command calculates (1) how much bias there must be in an estimate to invalidate/sustain an inference; (2) the impact of an omitted variable necessary to invalidate/sustain an inference for a regression coefficient.

pkonfound( est_eff, std_err, n_obs, n_covariates = 1, alpha = 0.05, tails = 2, index = "RIR", nu = 0, n_treat = NULL, switch_trm = TRUE, model_type = "ols", a = NULL, b = NULL, c = NULL, d = NULL, two_by_two_table = NULL, test = "fisher", replace = "control", to_return = "print" )

est_eff | the estimated effect (such as an unstandardized beta coefficient or a group mean difference) |
---|---|

std_err | the standard error of the estimate of the unstandardized regression coefficient |

n_obs | the number of observations in the sample |

n_covariates | the number of covariates in the regression model |

alpha | probability of rejecting the null hypothesis (defaults to 0.05) |

tails | integer whether hypothesis testing is one-tailed (1) or two-tailed (2; defaults to 2) |

index | whether output is RIR or IT (impact threshold); defaults to "RIR" |

nu | what hypothesis to be tested; defaults to testing whether est_eff is significantly different from 0 |

n_treat | the number of cases associated with the treatment condition; applicable only when model_type = "logistic" |

switch_trm | whether to switch the treatment and control cases; defaults to FALSE; applicable only when model_type = "logistic" |

model_type | the type of model being estimated; defaults to "ols" for a linear regression model; the other option is "logistic" |

a | cell is the number of cases in the control group showing unsuccessful results |

b | cell is the number of cases in the control group showing successful results |

c | cell is the number of cases in the treatment group showing unsuccessful results |

d | cell is the number of cases in the treatment group showing successful results |

two_by_two_table | table that is a matrix or can be coerced to one (data.frame, tibble, tribble) from which the a, b, c, and d arguments can be extracted |

test | whether using Fisher's Exact Test or A chi-square test; defaults to Fisher's Exact Test |

replace | whether using entire sample or the control group to calculate the base rate; default is the entire sample |

to_return | whether to return a data.frame (by specifying this argument to equal "raw_output" for use in other analyses) or a plot ("plot"); default is to print ("print") the output to the console; can specify a vector of output to return |

prints the bias and the number of cases that would have to be replaced with cases for which there is no effect to invalidate the inference

# using pkonfound for linear models pkonfound(2, .4, 100, 3)#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 60.3 % of the estimate would have to be due to bias. #> This is based on a threshold of 0.794 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 60 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 60). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.#>#>pkonfound(-2.2, .65, 200, 3)#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 41.732 % of the estimate would have to be due to bias. #> This is based on a threshold of -1.282 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 83 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 83). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.#>#>pkonfound(.5, 3, 200, 3)#> Robustness of Inference to Replacement (RIR): #> To sustain an inference, 91.549 % of the estimate would have to be due to bias. #> This is based on a threshold of 5.916 for statistical significance (alpha = 0.05). #> #> To sustain an inference, 183 of the cases with 0 effect would have to be replaced with cases at the threshold of inference (RIR = 183). #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460.#>#>pkonfound(-0.2, 0.103, 20888, 3, n_treat = 17888, model_type = "logistic")#> Conclusion: #> User-entered Table: #> Fail Success #> Control 2882 118 #> Treatment 17308 580 #> #> #> Note: Values have been rounded to the nearest integer. #> This may cause a little change to the estimated effect for the Implied Table. #> #> To sustain an inference for a negative treatment effect, you would need to replace 1 treatment success cases #> with null hypothesis cases (RIR = 1). #> This is equivalent to transferring 1 case from treatment success to treatment failure, #> as shown, from the Implied Table to the Transfer Table. #> Transfer Table: #> Fail Success #> Control 2882 118 #> Treatment 17309 579 #> #> For the Implied Table, we have an estimate of -0.200, with a SE of 0.103 and a t-ratio of -1.946. #> and a t-ratio of -0.200. #> RIR: #> RIR = 1#>#>pkonfound(2, .4, 100, 3, to_return = "thresh_plot")pkonfound(2, .4, 100, 3, to_return = "corr_plot")pkonfound_output <- pkonfound(2, .4, 200, 3, to_return = c("raw_output", "thresh_plot", "corr_plot") )#> Robustness of Inference to Replacement (RIR): #> To invalidate an inference, 60.557 % of the estimate would have to be due to bias. #> This is based on a threshold of 0.789 for statistical significance (alpha = 0.05). #> #> To invalidate an inference, 121 observations would have to be replaced with cases #> for which the effect is 0 (RIR = 121). #> #> See Frank et al. (2013) for a description of the method. #> #> Citation: Frank, K.A., Maroulis, S., Duong, M., and Kelcey, B. (2013). #> What would it take to change an inference? #> Using Rubin's causal model to interpret the robustness of causal inferences. #> Education, Evaluation and Policy Analysis, 35 437-460. #>#>#> Created 3 forms of output. To access type: #> #> pkonfound_output$raw_output #> pkonfound_output$thresh_plot #> pkonfound_output$corr_plotpkonfound_output$raw_output#> # A tibble: 1 x 8 #> action inference percent_bias_to… replace_null_ca… unstd_beta beta_threshhold #> <chr> <chr> <dbl> <dbl> <dbl> <dbl> #> 1 to_inv… reject_n… 60.6 121 2 0.789 #> # … with 2 more variables: omitted_variable_corr <dbl>, itcv <dbl>pkonfound_output$thresh_plotpkonfound_output$corr_plot# using pkonfound for a 2x2 table pkonfound(a = 35, b = 17, c = 17, d = 38)#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 13 treatment success #> cases with null hypothesis cases (RIR = 13). #> This is equivalent to transferring 9 cases from treatment success to treatment failure. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.278, with p-value of 0.051: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 26 29 #> #> RIR: #> RIR = 13#>#>pkonfound(a = 35, b = 17, c = 17, d = 38, alpha = 0.01)#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 9 treatment success #> cases with null hypothesis cases (RIR = 9). #> This is equivalent to transferring 6 cases from treatment success to treatment failure. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.835, with p-value of 0.011: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 23 32 #> #> RIR: #> RIR = 9#>#>pkonfound(a = 35, b = 17, c = 17, d = 38, alpha = 0.01, switch_trm = FALSE)#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 18 control failure #> cases with null hypothesis cases (RIR = 18). #> This is equivalent to transferring 6 cases from control failure to control success. #> For the User-entered Table, we have an estimated odds ratio of 4.530, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have an estimated odds ratio of 2.790, with p-value of 0.012: #> Transfer Table: #> Fail Success #> Control 29 23 #> Treatment 17 38 #> #> RIR: #> RIR = 18#>#>pkonfound(a = 35, b = 17, c = 17, d = 38, test = "chisq")#> Background Information: #> This function calculates the number of cases that would have to be replaced #> with no effect cases (RIR)to invalidate an inference made about the association #> between the rows and columns in a 2x2 table. #> One can also interpret this as switches from one cell to another, such as from #> the treatment success cell to the treatment failure cell. #> #> Conclusion: #> To invalidate the inference, you would need to replace 15 treatment success #> cases with null hypothesis cases (RIR = 15). #> This is equivalent to transferring 10 cases from treatment success to treatment failure. #> For the User-entered Table, we have a Pearson's chi square of 14.176, with p-value of 0.000: #> #> User-entered Table: #> Fail Success #> Control 35 17 #> Treatment 17 38 #> #> #> For the Transfer Table, we have a Pearson's chi square of 3.640, with p-value of 0.056: #> Transfer Table: #> Fail Success #> Control 35 17 #> Treatment 27 28 #> #> RIR: #> RIR = 15#>#>