STATISTICAL ASSESSMENT OF SAMPLING VARIABILITY AND CONFIDENCE INTERVAL COVERAGE USING BOOTSTRAP METHODS
DOI:
https://doi.org/10.69980/84g63842Keywords:
Bootstrap Resampling, Sampling Variability, Confidence Interval Coverage, Monte Carlo Simulation, Statistical InferenceAbstract
Bootstrap resampling is widely applied to statistical inference to estimate the sampling variability and derive confidence intervals. In small-scale studies, however, its reliability may be affected by the sample size, probability distribution and estimation uncertainty. The purpose of this study was to assess the sampling variability, estimation accuracy and empirical coverage of the confidence intervals from sample-based statistical powers in the different simulation conditions. The study is of quantitative, simulation type with secondary data that is publicly available from Zenodo. Thirty-six conditions—three probability distributions (normal, skewed, and ceiling/discrete), four effect sizes, and three different pilot sample sizes (10, 15, and 20 per group) were analysed. There were 400 stand-alone pilot simulations with 400 bootstrap replications per pilot simulation per condition. The statistical performance was evaluated by standard deviation, estimation bias, root mean squared error (RMSE) and empirical coverage of confidence intervals. Results indicated that sampling variability (standard deviation between pilots) decreased as the number of samples collected in a pilot increased (0.2856 to 0.2179). The RMSE of Bootstrap also decreased from 0.3281 to 0.2363, indicating better estimation accuracy. However, its empirical mean coverage was as low as 14.37% as opposed to the nominal level of 95%. These results highlight the need to consider the various sources of uncertainty and improve the calibration of confidence intervals to obtain more reliable statistical inference using the bootstrap in small-sample studies.
References
1.Aldawsari, A. (2023). Parametric predictive bootstrap and test reproducibility (Doctoral dissertation, Durham University).
2.Austin, P. C. (2022). Bootstrap vs asymptotic variance estimation when using propensity score weighting with continuous and binary outcomes. Statistics in Medicine, 41(22), 4426-4443.
3.Bottomley, C., Ooko, M., Gasparrini, A., & Keogh, R. H. (2023). In praise of Prais‐Winsten: An evaluation of methods used to account for autocorrelation in interrupted time series. Statistics in medicine, 42(8), 1277-1288.
4.Cheung, S. F., & Cheung, S. H. (2024). manymome: An R package for computing the indirect effects, conditional effects, and conditional indirect effects, standardized or unstandardized, and their bootstrap confidence intervals, in many (though not all) models. Behavior Research Methods, 56(5), 4862-4882.
5.da Silva, E. A., Rocha, D. S., Naves, E. L. M., & Rodrigues, T. B. (2026). Resampling-based power simulation in small samples: a Monte Carlo calibration study (reproducibility package) (Version v1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.20767954.
6.Falk, C. F., Vogel, T. A., Hammami, S., & Miočević, M. (2024). Multilevel mediation analysis in R: A comparison of bootstrap and Bayesian approaches. Behavior Research Methods, 56(2), 750-764.
7.Galavi, H., Mirzaei, M., Yu, B., & Lee, J. (2023). Bootstrapped ensemble and reliability ensemble averaging approaches for integrated uncertainty analysis of streamflow projections. Stochastic Environmental Research and Risk Assessment, 37(4), 1213-1227.
8.Greco, L., Luta, G., & Wilcox, R. (2024). On testing the equality between interquartile ranges: L. Greco et al. Computational statistics, 39(5), 2873-2898.
9.Hawinkel, S., Waegeman, W., & Maere, S. (2024). Out-of-sample R 2: Estimation and inference. The American Statistician, 78(1), 15-25.
10.Herrera-Casanova, R., Conde, A., & Santos-Pérez, C. (2024). Hour-ahead photovoltaic power prediction combining BiLSTM and Bayesian optimization algorithm, with bootstrap resampling for interval predictions. Sensors, 24(3), 882.
11.Innocenti, S., Matte, P., Fortin, V., & Bernier, N. (2022). Analytical and residual bootstrap methods for parameter uncertainty assessment in tidal analysis with temporally correlated noise. Journal of Atmospheric and Oceanic Technology, 39(10), 1457-1481.
12.Jané, M., Xiao, Q., Yeung, S., Ben-Shachar, M. S., Caldwell, A., Cousineau, D., ... & Feldman, G. (2024). Guide to effect sizes and confidence intervals. DOI: https://doi. org/10.17605/OSF. IO/D8C4G.
13.Jiang, R., Zhan, X., & Wang, T. (2023). A flexible zero-inflated poisson-gamma model with application to microbiome sequence count data. Journal of the American Statistical Association, 118(542), 792-804.
14.Johnson, M. R., Conrad, B. M., & Tyner, D. R. (2023). Creating measurement-based oil and gas sector methane inventories using source-resolved aerial surveys. Communications Earth & Environment, 4(1), 139.
15.Justus, V. L., Rodrigues, V. B., & Sousa, A. R. D. S. (2024). Bootstrap confidence intervals: A comparative simulation study. arXiv preprint arXiv:2404.12967.
16.Kluger, D. M., Lu, K., Zrnic, T., Wang, S., & Bates, S. (2025). Prediction-powered inference with imputed covariates and nonuniform sampling. arXiv preprint arXiv:2501.18577.
17.Lee, K., Ko, J., & Jung, S. (2024). Quantifying uncertainty in carbon emission estimation: Metrics and methodologies. Journal of Cleaner Production, 452, 142141.
18.Lin, S., Bun, M., Gaboardi, M., Kolaczyk, E. D., & Smith, A. (2024). Differentially private confidence intervals for proportions under stratified random sampling. Electronic Journal of Statistics, 18(1), 1455-1494.
19.Maitner, B. S., Halbritter, A. H., Telford, R. J., Strydom, T., Chacon, J., Lamanna, C., ... & Enquist, B. J. (2023). Bootstrapping outperforms community‐weighted approaches for estimating the shapes of phenotypic distributions. Methods in Ecology and Evolution, 14(10), 2592-2610.
20.Mayen, M. S. A., Nisha, S. N., Afrin, S., Ahammed, T., Chowdhury, M. A. B., & Uddin, M. J. (2024). Evaluating the current methodological practices and issues in existing literature in pooling complex surveys: a systematic review. BMC Medical Research Methodology, 24(1), 279.
21.Mokhtar, S. F., Yusof, Z. M., & Sapiri, H. (2023). Confidence intervals by bootstrapping approach: a significance review. Malaysian Journal of Fundamental and Applied Sciences, 19(1), 30-42.
22.Mokhtar, S. F., Yusof, Z. M., & Sapiri, H. (2024). Confidence interval estimating the mean of normal distribution and skewed distribution. Malaysian Journal of Fundamental and Applied Sciences, 20(5), 1124-1135.
23.Murphy, C., Tawn, J. A., & Varty, Z. (2025). Automated threshold selection and associated inference uncertainty for univariate extremes. Technometrics, 67(2), 215-224.
24.Nguinabé, J., Rockefeller, R., Khan, N. M., & Rodrigues, P. C. (2025). Bootstrap prediction intervals for the long short term memory (LSTM) recurrent neural network. Expert Systems with Applications, 284, 127728.
25.Rousselet, G., Pernet, C. R., & Wilcox, R. R. (2023). An introduction to the bootstrap: a versatile method to make inferences by using data-driven simulations. Meta-Psychology, 7.
26.Schader, L. M., Song, W., Kempker, R., & Benkeser, D. (2024). Don’t let your analysis go to seed: on the impact of random seed on machine learning-based causal inference. Epidemiology (Cambridge, Mass.), 35(6), 764.
27.Tibbe, T. D., & Montoya, A. K. (2022). Correcting the bias correction for the bootstrap confidence interval in mediation analysis. Frontiers in psychology, 13, 810258.
28.Tongmual, N., & Yonwilad, W. (2025). Bayesian Bootstrap Confidence Interval for Mean based on Small Sample Sizes. Malaysian Journal of Fundamental and Applied Sciences, 21(6), 2937-2944.
29.Williams, D. R. (2022). Learning to live with sampling variability: Expected replicability in partial correlation networks. Psychological Methods, 27(4), 606.
30.Zhu, B., Wan, C., Wang, P., & Chevallier, J. (2025). Interval forecasting of carbon price with a novel hybrid multiscale decomposition and bootstrap approach. Journal of Forecasting, 44(2), 376-390.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Dr. Sofia Laurent, Dr. Lukas Schneider, Dr. Emily Carter
This work is licensed under a Creative Commons Attribution 4.0 International License.
