<<< The Kappa coefficient : The measurement of Interrater agreement when the ratings are on categorical scales. The case of two raters. >>> *** Program: Kappa.exe Author : Philippe Bonnardel - philippe@bonnardel.com *** The kappa statistic is a frequently used measure of interobserver or inter-diagnostic agreement when the data are categorical. Cohen first proposed the kappa as a chance-corrected coefficient of agreement nominal scales. Suppose that each of a sample of n subjects is rated independently by the same two raters, with the ratings being on a categorical scale consisting of k categories. Consider the hypothetical example, in which each cell entry is the number of all subjects classified into one of k = 3 diagnostic categories by rater A and into another by rater B : Rater A Rater B Category 1 Category 2 Category 3 Total Category 1 25 3 5 33 Category 2 6 32 8 46 Category 3 1 0 20 21 Total 32 35 33 100 The Cohen's kappa is defined by : K = (Po - Pe) / (1 - Pe) where: Po is the observed agreement Pe is the chance-expected agreement For our example : Po = 0.770 Pe = 0.336 K = 0.654 s.e.0(k) = 0.069 z = 9.422 p-value = 0.0 s.e(k) = 0.062 For most purposes : Kappa Agreement ----- --------- [1.0 - 0.75] Excellent > 0.41 Good <= 0.41 poor 0.0 Ratings are independent < 0.0 Disagreement > agreement -1.0 The best disagreement ! *** The program kappa.exe is written to calculate : - the kappa statistic and kappas for individual categories - the weighted kappa statistic - The kappa standard error for testing the hypothesis that the underlying value of kappa is zero - The critical ratio and p-value. - The kappa standard error if kappa .NE. 0 *** The program kappa.exe accepts : -> 2 <= Categories <= 25 -> 32767 subjects in each cell of the contingency table *** Command line -> Kappa.exe filename.ext (By default the input file name is 'kappa.in') *** Cohen's weighted kappa You must write an input file like this ('filename.in'): Title "Interexaminer agreement in periondontal research" categories 10 data 0 8 1 1 0 0 0 0 0 0 10 63 35 7 0 0 1 0 0 0 3 28 41 15 10 0 0 0 0 0 0 5 20 21 11 3 0 0 0 0 0 1 2 8 15 1 2 1 0 0 0 0 0 0 3 2 5 3 0 0 0 1 0 1 3 3 2 2 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 1 0 0 2 1 0 0 0 0 0 0 0 0 1 0 1 weight 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 1 *** Cohen's kappa You must write an input file like this ('filename.in'): Title "Interexaminer agreement in periondontal research" categories 10 data 0 8 1 1 0 0 0 0 0 0 10 63 35 7 0 0 1 0 0 0 3 28 41 15 10 0 0 0 0 0 0 5 20 21 11 3 0 0 0 0 0 1 2 8 15 1 2 1 0 0 0 0 0 0 3 2 5 3 0 0 0 1 0 1 3 3 2 2 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 1 0 0 2 1 0 0 0 0 0 0 0 0 1 0 1 -> All results are in the output file 'filename.out'