Thanks Paul and Richard for your comments.....
To answer one question posed above, to provide some background, and
also to ask another question....
The background of this issue is that of the design of experiments for
a contact strategy.
The two groups are test and control.
The customers are assigned to the two groups by randomization along
their unique customer id.
I believe that the programmers just use the SAS rannor function.
The programmers have no information about any customer attribute ---
just unique customer id, some filters and a huge (millions) target
population.
It is only later that we go it, match the customer to their various
customer factors and check to see if the customers in the two groups
look alike.
Prior to this audit check, there was a problem in an experiment where
the contact strategy was to call or not to call customers. Due to
some misunderstanding, the control group was populated with people who
were also on do-not-call lists. Turns out that these are also people
of a certain age and a certain risk score. Which we only found out
later.
Hence the instigation of the audit functionality.
I would like to run another concept by all of you.
One of the ways that the bosses are thinking about this test for
"sameness" in the distributions is how it relates to a "cost
function". Think of the 2x2 matrix of results from a statistical
test, where the Type I error is in the top right cell (reject when the
null is true) and the Type II error is in the bottom left cell (accept
when the null is false) and the other two cells are correct choices.
They say that in testing, there is an implied cost function overlaid
on the test --- where each of those quadrants has a cost value of 1 --
we want to evaluate them all at the same level. But, what if,
minimizing Type II error is more im****tant than minimzing Type I
error. Then how should the test be constructed?
And I don't know how to respond.
These thought experiments leave me a little confused and even more
frustrated.
It is just a simple audit of a list pull process......
Any comments?
-jennifer.
|
