My background is mostly in Lean, so to expand my skill-set, I recently went through a really solid Lean Six Sigma training course at the UT-Arlington's Texas Manufacturing Assistance Center (TMAC). TMAC does a great job of developing your analytical skills, and this week I got to use one of my newfound skills--calculating Kappa.
What is Kappa?
It's a score that tells us how reliable our data collection system is when the data being collected is attribute/discrete data like pass/fail, good/bad, etc. Basically, we take two scorers (the folks who decide if something should be recorded as 'pass' or 'fail') and we have them score a sample of products twice. We want a minimum sample size of 40, with about half being good products and half being defective products. Once the samples are scored twice by each scorer, we then plug the results into a spreadsheet and do some calculations to get the Kappa score.
Why two scorers? Why two rounds of scoring each?
Having two scorers lets us see scoring variation between scorers. Having each scorer score the samples twice lets us see scoring variation for one individual across two rounds. If the scorers are consistent with each other, we feel good. If the scorers are consistent with themselves between the two rounds, we again feel good.
What is a good Kappa score?
When looking at the Kappa score between two scorers, if we get above 0.70 (out of 1.0), we've got a pretty good data collection system. When looking at the Kappa score for one scorer between two rounds, if we get above 0.85 (again out of 1.0), we've got a pretty good data collection system.
How does this fit into Lean/Six Sigma and management in general?
Whether we're leading a Lean/Six Sigma project or using data to guide us in everyday management, we want to know if our data is valid. That's what a Measurement System Analysis (MSA) is all about, and a Kappa calculation is just one way of doing a MSA if our data is attribute/discrete data.
Of course, even if we prove that our data is valid, we should never rely on data alone. We can surely use data to point us in the right direction to where problems are, but we should always go to the gemba and check out the gembutsu with our own eyes. Never let data get in the way of facts.
Where can I learn more?
There's a guide I recommend, The Lean Six Sigma Pocket Toolbook by Michael L. George, et al. Starting on pg. 100, you'll see a complete explanation for how to calculate Kappa.
