Randomization tests for small samples

Abstract

An advantage of randomization tests for small samples is that an exact P-value can be computed under an additive model. A disadvantage with very small sample sizes is that the resulting discrete distribution for P-values can make it mathematically impossible for a P-value to attain a particular degree of significance. We investigate a distribution of P-values that arises when several thousand randomization tests are conducted simultaneously using small samples, a situation that arises with microarray gene expression data. We show that the distribution yields valuable information regarding groups of genes that are differentially expressed between two groups: a treatment group and a control group. This distribution helps to categorize genes with varying degrees of overlap of genetic expression values between the two groups, and it helps to quantify the degree of overlap by using the P-value from a randomization test. Moreover, a statistical test is available that compares the actual distribution of P-values with an expected distribution if there are no genes that are differentially expressed. We demonstrate the method and illustrate the results by using a microarray data set involving a cell line for rheumatoid arthritis. A small simulation study evaluates the effect that correlated gene expression levels could have on results from the analysis.