Comparing the performance of randomization tests and classical non-parametric tests

Abstract

In the present world of data, there are plenty of instances where data does not adhere to the parametric assumptions. However, with the development of technology and computational facilities, Randomization Tests have become a popular alternative for these situations. Randomization tests also does not make any distributional assumptions but differ from classical non- parametric tests in many aspects. In this study, Wilcoxon Signed Rank test for both one sample and paired samples, Mann-Whitney test, Kruskal-Wallis test and Friedman test are being compared with respective Randomization Tests. The performance of the tests is compared for various sample sizes where 30 being the maximum sample size since the non-parametric tests are more useful when sample size is less than 30 and different variance conditions as well. The tests are compared for both skewed and symmetrically distributed data. Exponential and Gamma distributed data were generated as skewed data while Uniform and Normal distributed data are generated for symmetric data. The tests are being compared using type 1 error and power. Data were simulated using R software under each case and performed both tests for the same sample of data and obtained their type 1 error and power. Through this study, many cases were identified where randomization tests is more reliable than classical non- parametric tests in terms of type 1 error. For two skewed distributed unbalanced samples with heterogeneous variances, randomization tests are reliable since their type 1 error was around the pre-defined level while Mann-Whitey test is not reliable. Further, it could be found that randomization test is capable of protecting the type 1 error than Kruskal-Wallis test for large samples of non-normal symmetric and exponentially distributed data with heterogeneous variances. There were several situations of Friedman test performing meagerly than randomization test in terms of type 1 error. When considering the power values of the tests, randomization tests show almost equal robustness with classical non-parametric tests in most of the cases. However, for large sample sizes, randomization test has high power than Wilcoxon Signed Rank test (one sample). Moreover, when data are non-normal but symmetrically distributed, randomization test is more powerful than Kruskal-Wallis test according to the type 1 error. For symmetric data, randomization test is more powerful than Friedman test in both heterogeneous and homogenous samples. Hence, it is necessary to apply the two tests carefully as described in this study for more accurate analysis of data.