jenmoocat wrote:
>
> However, would you mind expanding on the idea of a "compound" null
> hypothesis?
Simple null hypothesis: population parameter exactly equals something,
or difference in two parameters equals zero. In this case, the
parameters are the probability mass functions for the two groups, and
the null hypothesis is that the mass functions are identical.
Compound null hypothesis: population parameter (or difference in
parameters) is in some range. This would translate in your case to the
difference in the two mass functions being less than some cutoff, or one
mass function being in a "neighborhood" of the other, where the
"diameter" of the "neighborhood" reflects some level of indifference or
tolerance ("good enough for government work").
Simple null hypotheses are popular because it's relatively easy to
derive the distribution of the test statistic under a simple hypothesis;
but in practice it's not often the case that you really expect a
parameter to exactly equal some target value, or two parameter to
exactly equal each other, and if there's any difference a sufficiently
large sample size will lead you to reject the null. I think that's what
you're experiencing with the chi-square test.
So what you need is a test who's null hypothesis is that the mass
function for group 1 is "close enough" to the mass function for group 2,
which means (a) you need a measure of closeness, (b) you need a
quantification of "close enough" and (c) you need a test that can handle
this type of null hypothesis.
> Because the idea of "the difference in the distributions is less than
> ___, where the difference is defined as some metric" is exactly what
> my bosses are proposing and they just want to utilize the chi-square
> distribution and kind of turn it on its head.
If by "turn it on it's head" you mean swap the null hypothesis for the
alternate hypothesis, I don't think that makes sense.
> Which is the thing that feels wrong to me.
> I guess I need some ammunition to be able to say --- you can't just
> see whether the calculated chi-square stat is less than the degrees of
> freedom.
> That doesn't prove that the distributions are close to one
> another.....
The two-sample Kolmogorov-Smirnov test
(http://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test)
defines the
difference in distributions in this case to be the maximum absolute
difference between the two CDFs. The two-sample Crame'r-von Mises test
(http://en.wikipedia.org/wiki/Cram%C3%A9r-von-Mises_criterion)
uses the
mean squared difference of the CDFs. As with most tests, the
distribution of the test statistic is derived for a simple null
hypothesis (that the two distributions are the same). I don't know if
anyone has come up with a distribution for either statistic under the
null hypothesis that the two distributions are "close".
I suppose you could pick either measure (maximum absolute difference or
mean squared difference between the CDFs) and simply declare a cutoff
for closeness. Thinking about it, I'm not sure whether you actually
need a statistical test. If your groups are full populations (not
samples from larger populations), then testing is not needed -- just
define a closeness measure, compute it and see how things turn out. If
you are sampling larger populations, then a test is needed.
/Paul
|
