BiostatKit / Agreement / Cohen's kappa

Cohen's Kappa Calculator

Enter how two raters classified the same subjects and get Cohen's kappa with its 95% confidence interval, the observed and chance agreement, a sentence you can paste into your manuscript, and the R code that reproduces the numbers.

Your data never leaves your browser. No sign-up, no ads.

Your agreement table

Pick the number of categories, then enter the counts.

Number of categoriesEach rater puts every subject in one category.

WeightingWeights give partial credit to near misses. Order the categories from lowest to highest first.

Counts of subjects. Rows are the ratings of rater 1, columns the ratings of rater 2.

Rater 1 (rows)
Rater 2 (columns)

Cat. 1

Cat. 2

Total

Cat. 1

Cat. 2

Total

100

Result

Agreement between two raters beyond chance.

Sentence for your manuscript

Written in AMA style from your numbers. Check it against your own study design before you submit.

Reporting checklist

GRRAS 2011 (reliability and agreement studies): describe the subjects, the raters and how they were chosen, whether raters worked independently, and report the agreement estimate with its confidence interval.
Report the observed agreement next to kappa, and the number of categories and weights used.
For ordered categories say why you chose linear or quadratic weights.

Methods wording. Agreement between two raters was calculated with BiostatKit [1] as Cohen's kappa [2], with the 95% CI from the large-sample standard error of Fleiss, Cohen, and Everitt.
[1] BiostatKit. Cohen's kappa calculator. Accessed [date]. [URL]
[2] Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20(1):37-46.

Reproduce it in R

The script is filled in with your numbers. Paste it into R or RStudio and compare.

R 4.x · base R only

This calculator was checked against Python (SciPy and statsmodels, which implements the same kappa and standard error) on 2 reference cases and matched to 4 significant digits. See the verification log.

How to read Cohen's kappa

What it is

Kappa measures how much two raters agree beyond what chance alone would produce. Kappa = 1 is perfect agreement, 0 is the agreement expected by chance, and below 0 is worse than chance.

How this page computes it

κ = (p_o - p_e) / (1 - p_e) with p_o the observed agreement (the diagonal) and p_e the agreement expected from the row and column totals.

With weights, each cell has a weight w (1 on the diagonal; linear: 1 - |i - j| / (K - 1); quadratic: 1 - ((i - j) / (K - 1))²), and p_o and p_e are weighted sums (Cohen, 1968). The standard error is the large-sample formula of Fleiss, Cohen, and Everitt (1969), and the 95% CI is κ ± 1.96 x SE. An upper limit above 1 is shown as 1.

Assumptions

Two raters classify each subject independently into the same categories.
Subjects are independent of each other.
The large-sample CI needs a reasonable number of subjects. With few subjects treat it as rough.

Common mistakes

Reading the Landis and Koch labels (0.61-0.80 substantial, 0.81-1.00 almost perfect) as a test. They are conventions, not cut-offs (Landis and Koch, 1977).
Reporting kappa alone. When one category is very common, kappa can be low although agreement is high. Show the observed agreement and the table.
Using weights for unordered categories, or unweighted kappa for a scale where near misses should count less.
Using kappa for continuous measurements. Use the intraclass correlation or a Bland-Altman analysis (both are on this site).

Frequently asked questions

Weighted or unweighted kappa?

Use unweighted kappa when the categories have no order (diagnosis A, B, C). Use linear or quadratic weights when they are ordered (mild, moderate, severe), so that a one-step miss counts less than a two-step miss.

Why is my kappa low when most ratings agree?

Kappa corrects for chance agreement, and when almost all subjects fall in one category the chance agreement is already high. Report the observed agreement and the counts as well.

What is a good kappa?

Landis and Koch suggested 0.61 to 0.80 as substantial and above 0.80 as almost perfect, but they called their own bands arbitrary. Judge kappa against what your field needs and report its confidence interval.

What about more than two raters?

This calculator is for exactly two raters rating the same subjects. For three or more raters use Fleiss' kappa or the intraclass correlation.

Is my data stored or sent anywhere?

No. All calculations run in your browser tab. Nothing is uploaded, and the page works offline once loaded.