Hypothesis testing permits us to compare two groups of items and determine if there is a significant difference or not. There are many types of hypothesis tests depending on the specific question, type of data, and what is or is not known when designing the test.
One of the most often asked question concerning nearly any testing is the sample size. There are times, although sometimes rare, when we have sufficient samples to run the test properly – statistically speaking. Whether we have ample samples or not, we should calculate the sample size related to the desired confidence.
The α error that can be tolerated should be set by local policy (if not, select a reasonable number at or below 0.4, common is 0.1 which is related to a 90% Type-1 confidence.) The next bit of information you need is how much of a difference is desired that the test actually detects. For example, if we are considering heights of people (for some reason) we may want to know if a specific group’s average height is at least 6 inches less than the overall population. Thus if the two groups of data are at least 6 inches different the test is designed to be able to detect that difference. The last bit of information we need is the variation of the data (standard deviation either known or estimated).
Given that we’ve selected a particular hypothesis test to run, and we’ve collected the three bits of information we need, we can calculate the minimum number of samples needed for the hypothesis test. Let’s assume we’re using a Z-test for a hypothesis about the means.
$$ \large\displaystyle Z=\frac{\bar{X}-\mu }{{}^{\sigma }\!\!\diagup\!\!{}_{\sqrt{n}}\;}$$
Z as you recall the test statistic and compared to a critical value for the decision at hand.
The X-bar minus μ is the data average difference from the population mean. In our planning stage, this is the amount of difference we would like to be able to detect with the test. For the height example above, this would be 6 inches. Let’s change the notation by stating the difference as E.
$$ \large\displaystyle E=\bar{X}-\mu $$
E is the minimum difference of interest. Next, let’s isolate the n term with some simple algebra.
$$ \large\displaystyle \sqrt{n}=\frac{Z\sigma }{E}$$
and, square both sides to finish.
$$ \large\displaystyle n=\frac{{{Z}^{2}}{{\sigma }^{2}}}{{{E}^{2}}}$$
Continuing our example of heights, let’s use E = 6, σ=6, and we’d like at least 95% confidence making α=0.05 and Z=1.96. Insert the values a little math, and we find the sample size.
$$ \large\displaystyle n=\frac{{{1.96}^{2}}{{6}^{2}}}{{{6}^{2}}}=3.84\approx 4$$
In this simple example, the standard deviation and difference of interest at the same and cancel out making the calculation very easy. We need a sample of four from the group to determine if that group’s average height is 6 inches lower or not than the population.
If we wanted higher confidence notice that Z increases rapidly. Also, if we wanted to detect less of a difference, again sample size increases rapidly as we want to detect a smaller difference, approaching infinity as the difference approaches zero. And, if the population has a higher variation than this example, again the sample size goes up.
Ideally, we always want to work with populations with very small amount of variation, relative low confidence (although many argue for at least 80 to 95% confidence as acceptable), and the desire to detect very large differences. Of course, this is often not the case. This sample size formula permits you to work with your team to balance the test cost, sample size, confidence, and sensitivity (E) of the planned test.
One can start with any hypothesis test and with some ‘rearrangement’ determine the sample size formula for that test. For those preparing for the CRE exam to work though a few of these and have the formulas in your notes.
Related:
Hypothesis Testing (article)
Paired-Comparison Hypothesis Tests (article)
Sample Size – success testing (article)
Leave a Reply