BiostatKit / Agreement / Intraclass correlation
Intraclass Correlation (ICC) Calculator
Paste the ratings (one row per subject, one column per rater) and get the intraclass correlation coefficient with its 95% confidence interval, all six forms in one table, a sentence you can paste into your manuscript, and the R code that reproduces the numbers.
Your data never leaves your browser. No sign-up, no ads.Result
Reliability of measurements made by several raters.
Sentence for your manuscript
Written in AMA style from your numbers. Check it against your own study design before you submit.
Reporting checklist
- GRRAS 2011 (reliability and agreement studies): describe the subjects and the raters and how they were chosen, whether raters worked independently, and report the ICC with its confidence interval.
- Name the ICC form you used: model (one-way, two-way random, two-way mixed), type (single rater or mean of raters) and definition (absolute agreement or consistency). Koo and Li (2016) ask for all three.
- Judge reliability by the confidence interval, not by the point estimate alone.
Methods wording. Reliability was calculated with BiostatKit [1] as the intraclass correlation coefficient [2], using the form named in the results, with the 95% CI from the F distribution.
[1] BiostatKit. Intraclass correlation (ICC) calculator. Accessed [date]. [URL]
[2] Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-428.
Reproduce it in R
The script is filled in with your numbers. Paste it into R or RStudio and compare.
This calculator was checked against Python (statsmodels mean squares and SciPy F distribution; the six point estimates also equal the worked example in Shrout and Fleiss) on 2 reference cases and matched to 4 significant digits. See the verification log.
How to read the intraclass correlation
What it is
The ICC is the share of the total variation in the ratings that comes from real differences between subjects rather than from the raters. 1 means the raters give the same values, 0 means the ratings tell you nothing about the subjects.
How this page computes it
It splits the variation into mean squares: MSB between subjects, MSJ between raters, MSE residual (two-way) and MSW within subjects (one-way). With n subjects and k raters:
ICC(1,1) = (MSB - MSW) / (MSB + (k - 1) MSW)
ICC(2,1) = (MSB - MSE) / (MSB + (k - 1) MSE + k (MSJ - MSE) / n)
ICC(3,1) = (MSB - MSE) / (MSB + (k - 1) MSE)
The mean-of-raters forms are ICC(1,k) = (MSB - MSW) / MSB, ICC(2,k) = (MSB - MSE) / (MSB + (MSJ - MSE) / n) and ICC(3,k) = (MSB - MSE) / MSB. The 95% CIs use the F distribution (Shrout and Fleiss, 1979; for the two-way random form McGraw and Wong, 1996, with Satterthwaite degrees of freedom).
Which form to pick
One-way: different raters for different subjects. Two-way random: the raters stand for any raters, so absolute agreement counts a rater who scores high all the time as a disagreement. Two-way mixed: only these raters matter and a constant offset between raters is ignored (consistency). Choose the single-rater unit if one rater's score will be used later, the mean unit if the average will be.
Assumptions
- Ratings are numbers on an interval scale, and every rater rated every subject (no missing values).
- Subjects are independent, and raters worked independently of each other.
- Errors are roughly normal with similar spread across subjects and raters.
- Subjects cover the range of values you will meet in practice. A narrow range lowers the ICC.
Common mistakes
- Using Pearson's correlation for agreement. It ignores a constant offset between raters.
- Reporting "ICC = 0.8" without the form, so nobody knows which of the six it is.
- Judging by the point estimate. Koo and Li (2016) read the bands (below 0.50 poor, 0.50-0.75 moderate, 0.75-0.90 good, above 0.90 excellent) from the confidence interval.
- Using the ICC for yes/no or category ratings. Use Cohen's kappa.
- Too few subjects or raters. Koo and Li suggest at least 30 varied subjects and 3 raters.
Frequently asked questions
Which ICC should I use?
If different raters rated different subjects use one-way. If the same raters rated everyone and you want the result to hold for other raters too, use two-way random with absolute agreement. If only these raters matter, use two-way mixed with consistency. Then pick single rater or mean of raters according to how the scores will be used.
What is a good ICC?
Koo and Li suggested below 0.50 poor, 0.50 to 0.75 moderate, 0.75 to 0.90 good and above 0.90 excellent, read from the 95% confidence interval. Judge the lower limit against what your field needs.
Why is my interval so wide, or even below zero?
With few subjects the F-based interval is wide, and a lower limit below zero means the data cannot rule out agreement no better than chance. More subjects narrow it.
ICC, kappa or Bland-Altman?
Use the ICC for numeric ratings by two or more raters, Cohen's kappa for categories, and a Bland-Altman analysis to show how far two measurement methods differ in the original units.
Is my data stored or sent anywhere?
No. All calculations run in your browser tab. Nothing is uploaded, and the page works offline once loaded.
Related calculators: Cohen's kappa · Bland-Altman analysis