Data layout
Use the same layout as the Gage R&R: part; appraiser; trial 1; trial 2; …. Add one row per part with the appraiser name Standard (or Reference) and the correct answer. Categories can be any text, such as OK / NOK, Pass / Fail or grades 1 to 5.
What is calculated
- Within appraisers: parts where all trials of one appraiser agree, with Fleiss' kappa across trials.
- Each appraiser vs standard: parts where all trials match the standard, with Cohen's kappa of all ratings against the standard.
- Between appraisers: parts where everybody agrees on every trial.
- 95 % confidence intervals use the exact Clopper-Pearson method.
Frequently asked questions
›What is an attribute agreement analysis?
It is the measurement system analysis for inspections that give a category instead of a number: pass / fail, go / no-go, or defect grades. It checks whether inspectors agree with themselves, with each other and with the known correct answer.
›What is kappa?
Kappa measures agreement corrected for the agreement you would get by chance. 1 means perfect agreement, 0 means no better than chance. AIAG treats kappa above 0.75 as good and below 0.40 as poor.
›How should I set up the study?
Use at least 30 parts, roughly half good and half bad including borderline parts, 2 or 3 appraisers and 2 or 3 trials in random order without the appraisers knowing which part is which. A known standard for every part makes the study much more useful.
›What is the difference between false accept and false reject?
A false accept (miss) is a bad part rated good, it reaches the customer. A false reject (false alarm) is a good part rated bad, it costs scrap or rework. AIAG guidelines are a miss rate below 2 % and a false alarm rate below 5 %.