FACTORS INFLUENCING TEST RELIABILITY

1.      The greater the number of items, the more accurate the test.  The respondents’ mental set for accuracy is important for reliability.  That is, variation in incentive or effort are important.  Perseverations from previous mental or emotional experiences are important.

2.      On the whole, the longer the test administration time, the greater the accuracy.  Stability may decline if tests are too long.

3.      The narrower the range of difficulty of items, the greater the reliability.  Items of moderate difficulty are preferred over easy or hard items.

4.      Interdependent items are those which require a correct answer on one item before it is possible to obtain a correct answer on others.  Such grouped items tend to reduce the reliability.

5.      The more systematic or “objective” the scoring, the greater the reliability coefficient.  Error due to mis-scored items reduces accuracy.

6.      The greater the probability of achieving success by chance (guessing), the lower the reliability.

7.      The more homogeneous the material, the greater the reliability.

8.      Reliability is affected by the extent to which individuals have similar characteristics.  Restricted range of characteristics in your sample can result in low reliability if there is no variance.  If there is variance, reliability can be increased.

9.      Trick questions lower the accuracy.  Subtle factors leading to misinterpretation of the test item lead to unreliability.

10.  Speed of work on test influences accuracy.  Some test-takers are set for speed and some are not.  Some test-takers distribute their time properly; some do not.

11.  Distractions have some effect on accuracy, although those effects can be overrated.  Accidents, like breaking a pencil or finding a defective test blank, are incidental factors.  The respondents attention to the task may be limited by illness, worry, or excitement.  These can affect accuracy although not always to the extent that most people think.

12.  Reliability generally decreases when there is intervening time between tests.  Delayed posttests are given for the purposes of establishing validity, not reliability.

13.  Cheating may be a factor in lowering accuracy or stability.

14.  Position of the individual on the learning curve for the tasks of the test may be important. (restriction of range)

INTERPRETATION OF CORRELATION COEFFICIENTS

1.      When may we call a coefficient “high” or “low?”
Stable coefficients from .00 to   .20 = negligible correlation
“”                         “”      .20 to   .40 = low degree of correlation
“”                         “”      .40 to   .60 = moderate degree of correlation
“”                         “”      .60 to   .80 = marked degree of correlation
“”                         “”      .80 to 1.00 = high degree of relation

2.      How high must a correlation be to be regarded as “satisfactory?”
The function of a coefficient of correlation is to measure the degree of association between two variables. In some situations a correlation of .00 might be satisfactory, and in others a correlation of .90 might be regarded as unsatisfactory. The coefficient stands merely as a statement of fact.

3.      Does correlation imply a causal relationship between two traits studied?
NO!

4.      Does the correlation coefficient indicate the percentage of agreement between the two traits? Does a coefficient of .20 mean 20 percent agreement?
NO! However, from the coefficient we can obtain a statement regarding the degree of overlap between the two variables. This is done by squaring the coefficient.

5.      Does a knowledge of the coefficient of correlation between two traits enable us to predict one from the other?
YES, but the relationship between the size of a correlation coefficient and its predictive value is not directly proportional. The lower correlations are of almost no value in prediction; the moderate ones only slightly better; and the marked coefficients are somewhat, but not very much better. Only as we advance into the high correlation range do the predictive values rise to usable levels. A statement of predictive efficiency can be found by the following formula:

                                                                  100 (1 -  (1-r2)1/2)

6.      Is there a direct arithmetical relationship between the size of a correlation and its value?  Is a coefficient of .75 three times as good as .25?
NO!  A statement can be made more accurate by looking at the squares of the correlation coefficients. The square of .25 is .0626, while the square of .75 is .5625. On this basis, a coefficient of .75 is nine times, not three times, better than a correlation of .25.

 ROC curve 분석 + construct validity 검증 연구 참고문헌(첨부 파일)

Psychometric properties and construct validity of the Obsessive–Compulsive Inventory—Revised: Replication and extension with a clinical sample  

* 위 연구는 cur-off score 결정방식을 Youden's index를 사용하지 않고,  sensitivity와 speciticity의 valance를 기준으로 하였음.

Jonathan S. Abramowitz a,*, Brett J. Deacon b

HLM : http://statlab.stat.yale.edu/help/workshops/HLMworkshop/statlab_hlm_intro_0407.pdf