Sorry, I didn't explain it sufficiently. My sup****t model is very
qualitative. I have a dozen dimensions of a decision of which I predict
that
performing well on these dimensions will contribute to the quality of a
person's decision.
I test this sup****t model also quite qualitatively (a more empirical
approach would indeed be much better but for now I first need to make a
qualitative *****sment of the sup****t model's validity). What I do is I
have
descriptions of decisions and examine these decisions qualitatively to see
whether they make sense if I look at them from the perspective of my
sup****t
model (i.e., I make a qualitative *****sment whether they took into
account
certain factors/dimensions and whether this contributed to the quality of
the decision. There are multiple researchers doing this test to be able to
test for intterrater-reliability).
This test, I call a "test of validity". However, someone suggested to me
that the terminology of validity is not appropriate in this context. I'm
not
sure myself whether it is justified to talk of "validity" for a sup****t
model as I described it.
What do you think, may I call this a test of validity?
"Greg Heath" wrote ...
> On Jul 23, 1:48 pm, "Burger" <burger1...@[EMAIL PROTECTED]
> wrote:
> > Hi All,
> >
> > I have developed a sup****t model / methodology to analyse certain
decisions.
> > To examine if it works I have applied it to a certain number of
decisions.
> > I call this a "test of validity".
> >
> > Now I got the remark from someone that a sup****t model does not have
such a
> > thing as "validity". I guess he's right and its more about whether the
> > sup****t model makes sense if applied in practice and whether it is
useful.
> > But couldn't I still call this validity? I don't see a better term; I
want
> > to use a technical/formal term.
> >
> > What do you think?
>
> In the world of neural networks, data is
> partitioned into design (in-sample) and
> test (out-of-sample) sets. The design set
> is further partitoned into a training set
> and a validation set.
>
> The training design set is used to estimate
> model parameters.
>
> The validation design set is used to estimate
> performance adequately enough to help make a
> choice between competing candidate models.
>
> Finally, the test set is used to obtain an
> unbiased estimate of performance on nondesign
> data.
>
> I don't know how much this train/validation/test
> partition concept has spilled into the nonneural
> model community. So you really have to search
> previously written references in your field.
>
> Hope this helps.
>
> Greg
>
|
