SIMULATION-BASED ASSESSMENT OF STATISTICAL POWER ESTIMATION FOR NONPARAMETRIC HYPOTHESIS TESTING
DOI:
https://doi.org/10.69980/twcd3g90Keywords:
Monte Carlo Simulation, Statistical Power Estimation, Nonparametric Hypothesis Testing, Bootstrap Resampling, Mann–Whitney U TestAbstract
The size of a study sample, and the nature of the distribution of the population under study, play a crucial role in hypothesis testing. The accuracy and reliability of the simulation-based statistical power estimation for nonparametric hypothesis testing was assessed by Monte Carlo simulations. Normal, skewed log normal and bounded discrete distributions were simulated, including standardized effect sizes between 0.0 to 0.9 and 10, 15 and 20 pilot observations per group. Power estimations for bootstrap-based and parametric were compared with reference statistical power, using repeated Mann–Whitney U tests. Bias, standard deviation, and root mean square error (RMSE) and coverage of confidence interval were used to evaluate the estimation performance. Results showed a general bootstrap bias of 0.060, and RMSE of 0.280 and a parametric RMSE of 0.299. A comparison of the accuracy of bootstrap estimation under skewed distributional conditions showed that it performed comparatively well. The bootstrap RMSE decreased from 0.328 to 0.236 with this increase in sample size (n=20) for the pilot sample, suggesting better estimation reliability. But the empirical coverage of the confidence intervals is only 0.144 while the nominal coverage is 0.950, with the empirical coverage being well below the nominal. The study finds that distributional properties and sample size affect simulation-based power estimation. Better uncertainty estimation techniques are needed to enhance the statistical confidence and aid in rational planning of the sample size in nonparametric studies.
References
1. Agronah, M. (2025). A Novel Approach for Simulation-Based Power Estimation and Joint Modeling of Microbiome Counts (Doctoral dissertation).
2. Altinbas, B. C., Çalık, K. Y., Erdöl, E. K., Kırkbir, İ. B., Güner, S. G., Tezel, M., ... & Bulut, H. K. (2025). The effect of simulation-based laboratory training on undergraduate nursing students’ clinical skill, satisfaction, and self-confidence. BMC nursing, 24(1), 1322.
3. Attar, A., Babaee, M., Raissi, S., & Nojavan, M. (2024). Airside Optimization Framework Covering Multiple Operations in Civil Airport Systems with a Variety of Aircraft: A Simulation-Based Digital Twin. Systems, 12(10), 394.
4. Awan, J., & Wang, Z. (2025). Simulation-based, finite-sample inference for privatized data. Journal of the American Statistical Association, 120(551), 1669-1682.
5. Bahçecitapar, M., & Akdur, H. T. K. (2022). Sample Size Estimation of Nonparametric Tests with Ordered Alternatives for Longitudinal Data in Randomized Complete Block Designs. Gazi University Journal of Science Part A: Engineering and Innovation, 9(4), 367-377.
6. Cadirci, M. S. (2025). Non-parametric goodness-of-fit tests using Tsallis entropy measures. Entropy, 27(12), 1210.
7. Chalmers, R. P. (2025). Spower: A general-purpose Monte Carlo simulation power analysis program. Behavior Research Methods, 57(12), 348.
8. Chatterjee, A., Niu, Z., & Bhattacharya, B. B. (2024). A kernel-based conditional two-sample test using nearest neighbors (with applications to calibration, regression curves, and simulation-based inference). arXiv preprint arXiv:2407.16550.
9. Chiang, K. S., Chang, Y. M., Liu, H. I., Lee, J. Y., Jarroudi, M. E., & Bock, C. H. (2024). Survival analysis as a basis for testing hypotheses when using quantitative ordinal scale disease severity data. Phytopathology®, 114(2), 378-392.
10. da Cruz, P. A. S. (2025). Backtesting Expected Shortfall: Historical Simulation Based Analysis of a US Diversified Portfolio.
11. da Silva, E. A., Rocha, D. S., Naves, E. L. M., & Rodrigues, T. B. (2026). Resampling-based power simulation in small samples: a Monte Carlo calibration study (reproducibility package) (Version v1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.20767954.
12. Delaunoy, A., Hermans, J., Rozet, F., Wehenkel, A., & Louppe, G. (2022). Towards reliable simulation-based inference with balanced neural ratio estimation. Advances in Neural Information Processing Systems, 35, 20025-20037.
13. Dennis, T., Rubin-McGregor, J., & O’Connell, M. J. (2025). Power analysis for zero-inflated Poisson and negative binomial generalized linear models using Monte Carlo simulation. Journal of Statistical Computation and Simulation, 95(4), 868-885.
14. Elgehady, H. S., El-Arishy, S. M., & El-Atfy, E. S. (2025). A Novel Hypothesis Testing for EBUC (mgf) Utilizing Laplace Transform Techniques with Practical Applications. Statistics, Optimization & Information Computing, 14(3), 1419-1439.
15. Fenteng, K. D. (2025). Power Analysis of Combined Nonparametric Tests: Fligner–Wolfe and Wald–Wolfowitz Approaches (Master's thesis, North Dakota State University).
16. Furberg, J. F., Andersen, P. K., Scheike, T., & Ravn, H. (2024). Simulation‐based sample size calculations of marginal proportional means models for recurrent events with competing risks. Pharmaceutical statistics, 23(5), 687-708.
17. Hidayat, K., Fernandes, A. A. R., Iriany, A., Solimun, S., Hidayatulloh, M. Z., & Junianto, F. H. (2025). Nonparametric Path Modeling with Double Resampling for Waste Economic Value Utilization: Simulation-Based Performance Comparison. CAUCHY: Jurnal Matematika Murni dan Aplikasi, 10(2), 1233-1250.
18. Irmer, J. P., Klein, A. G., & Schermelleh-Engel, K. (2024). Model-implied simulation-based power estimation for correctly specified and distributionally misspecified models: applications to nonlinear and linear structural equation models. Behavior research methods, 56(8), 8955-8991.
19. Kamoljitprapa, P., Sookkhee, S., & Polsen, O. (2024, August). Statistical analysis for genome data based on multiple SNPs using kernel machine based test. In 2024 Research, Invention, and Innovation Congress: Innovative Electricals and Electronics (RI2C) (pp. 262-266). IEEE.
20. Kelter, R. (2024). The Bayesian simulation study (BASIS) framework for simulation studies in statistical and methodological research. Biometrical Journal, 66(1), 2200095.
21. Martin Jr, F. (2023). An Experimental Mixed Methods Pilot Study for US Army Infantry Soldiers-Higher Levels of Combined Immersion and Embodiment in Simulation-Based Training Capabilities Show Positive Effects on Emotional Impact and Relationships to Learning Outcomes.
22. Mozaffari, A. (2025). Robust nonparametric inference on manifold spaces.
23. Ngurah, S. P. I. G. (2025). Evaluation of Fast Food Promotion Effectiveness Using a Hybrid Approach of Robust Non-Parametric Methods and Monte Carlo Simulation Based on Market Size Segmentation. Jurnal Akuntansi, Ekonomi dan Manajemen Bisnis, 13(1), 63-68.
24. Nilsson, P., & Eckert, A. (2024). Design principles for simulation-based learning of hypothesis testing in secondary school. Mathematical thinking and learning, 26(4), 359-381.
25. Petty, M. (2025). Trend Detection in Simulation-Based Training Performance Scores Obtained Under Varying Experimental Conditions. SAGE Publications Ltd. Petty, M. (2025). Trend Detection in Simulation-Based Training Performance Scores Obtained Under Varying Experimental Conditions. SAGE Publications Ltd.
26. Pulecio-Díaz, J. (2025). Curing sustainability assessment in concrete pavements: A 20-Year simulation-based analysis in urban road contexts. Sustainability, 17(12), 5299.
27. Riesthuis, P., Otgaar, H., & Bücken, C. (2025). Ready to ROC? A tutorial on simulation-based power analyses for null hypothesis significance, minimum-effect, and equivalence testing for ROC curve analyses. Behavior Research Methods, 57(4), 120.
28. Sadatsafavi, M., & Petkau, J. (2024). Non‐parametric inference on calibration of predicted risks. Statistics in Medicine, 43(18), 3524-3538.
29. Suchar, V. A., & Nardin, L. G. (2022). Simulation-Based Assessment of the Effectiveness of Tests for Stationarity. Statistics, 10(3), 670-683.
30. Verdier, H., Laurent, F., Cassé, A., Vestergaard, C. L., Specht, C. G., & Masson, J. B. (2023). Simulation-based inference for non-parametric statistical comparison of biomolecule dynamics. PLOS Computational Biology, 19(2), e1010088.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Dr. Claire Thompson, Dr. Thomas Berger, Dr. Hannah Mitchell
This work is licensed under a Creative Commons Attribution 4.0 International License.
