Burger wrote:
> Sorry, I didn't explain it sufficiently. My sup****t model is very
> qualitative. I have a dozen dimensions of a decision of which I predict
that
> performing well on these dimensions will contribute to the quality of a
> person's decision.
> I test this sup****t model also quite qualitatively (a more empirical
> approach would indeed be much better but for now I first need to make a
> qualitative *****sment of the sup****t model's validity). What I do is I
have
> descriptions of decisions and examine these decisions qualitatively to
see
> whether they make sense if I look at them from the perspective of my
sup****t
> model (i.e., I make a qualitative *****sment whether they took into
account
> certain factors/dimensions and whether this contributed to the quality
of
> the decision. There are multiple researchers doing this test to be able
to
> test for intterrater-reliability).
> This test, I call a "test of validity". However, someone suggested to me
> that the terminology of validity is not appropriate in this context. I'm
not
> sure myself whether it is justified to talk of "validity" for a sup****t
> model as I described it.
> What do you think, may I call this a test of validity?
>
>
> "Greg Heath" wrote ...
>> On Jul 23, 1:48 pm, "Burger" <burger1...@[EMAIL PROTECTED]
> wrote:
>>> Hi All,
>>>
>>> I have developed a sup****t model / methodology to analyse certain
> decisions.
>>> To examine if it works I have applied it to a certain number of
> decisions.
>>> I call this a "test of validity".
>>>
>>> Now I got the remark from someone that a sup****t model does not have
> such a
>>> thing as "validity". I guess he's right and its more about whether the
>>> sup****t model makes sense if applied in practice and whether it is
> useful.
>>> But couldn't I still call this validity? I don't see a better term; I
> want
>>> to use a technical/formal term.
>>>
>>> What do you think?
>> In the world of neural networks, data is
>> partitioned into design (in-sample) and
>> test (out-of-sample) sets. The design set
>> is further partitoned into a training set
>> and a validation set.
>>
>> The training design set is used to estimate
>> model parameters.
>>
>> The validation design set is used to estimate
>> performance adequately enough to help make a
>> choice between competing candidate models.
>>
>> Finally, the test set is used to obtain an
>> unbiased estimate of performance on nondesign
>> data.
>>
>> I don't know how much this train/validation/test
>> partition concept has spilled into the nonneural
>> model community. So you really have to search
>> previously written references in your field.
>>
>> Hope this helps.
>>
>> Greg
>>
>
> You are facing a similar challenge to what I am facing with my
disertation. After having gone round and round with my mentor, it seems
to come down to the fact that a model must be measured against ground
truth. The big question is what is ground truth for your model. As it
was explained to me there are multiple levels the top two being
1. Direct mapping of model to truth
2. Judgment form some form of authority
In my case I am using a survey to gather ground truth and then will
compare/map my model to the survey.
Another avenue to explore is where you think the work falls i.e.
Design science or behavioral science. If it is design science, the goal
is to discover utility where as if it behavioral science the goal is to
discover truth. See MISQ 28-1 March 2004 Design science in information
systems research, Heavner et al.
Bob Nolker
|
