To Sample or Not to Sample

Image courtesy of Stanford School of Medicine

I'm sitting in Six Sigma Black Belt training this week, learning all about two-sample t-tests, ANOVA, and other statistical analysis techniques.  One thing I noticed is that these techniques are based on sampling.  Basically, you collect data based on a sample, not the whole population.  An example from a hospital would be randomly picking 10 patients from a census of 100 and looking at their infection rates.

Obviously, data from a sample is not as thorough as that of a population, but often it's thorough enough to be statistically reliable.  The benefit of sampling, of course, is that we don't have to go through the time and expense of collecting data for the entire population.  However, thanks to powerful database software available to us in healthcare and pretty much any industry nowadays, we can easily pull all the data for all the patients in our system, at virtually no marginal cost.  This begs the question--why bother with sampling if we already have the population data?

I guess we wouldn't, unless there was some added value in sampling beyond the data that we gather.  If we're sampling by just pulling data out of a database, then there's probably not much value beyond the data.  But, if we're sampling by directly observing a process, then there's a lot of additional value:  we see the process with our own eyes, we get direct feedback from those involved with the process, we often get to directly hear the voice of the customer (the patient), and we get the opportunity to collect data that we didn't even know was relevant by looking at a database.

So, basically, it's not a question of "to sample or not to sample" but "to go & see or to not go & see."

Measurement System Analysis for Attribute/Discrete Data

My background is mostly in Lean, so to expand my skill-set, I recently went through a really solid Lean Six Sigma training course at the UT-Arlington's Texas Manufacturing Assistance Center (TMAC).  TMAC does a great job of developing your analytical skills, and this week I got to use one of my newfound skills--calculating Kappa.

What is Kappa?

It's a score that tells us how reliable our data collection system is when the data being collected is attribute/discrete data like pass/fail, good/bad, etc.  Basically, we take two scorers (the folks who decide if something should be recorded as 'pass' or 'fail') and we have them score a sample of products twice.  We want a minimum sample size of 40, with about half being good products and half being defective products.  Once the samples are scored twice by each scorer, we then plug the results into a spreadsheet and do some calculations to get the Kappa score.

Why two scorers?  Why two rounds of scoring each?

Having two scorers lets us see scoring variation between scorers.  Having each scorer score the samples twice lets us see scoring variation for one individual across two rounds.  If the scorers are consistent with each other, we feel good.  If the scorers are consistent with themselves between the two rounds, we again feel good.

What is a good Kappa score?

When looking at the Kappa score between two scorers, if we get above 0.70 (out of 1.0), we've got a pretty good data collection system.  When looking at the Kappa score for one scorer between two rounds, if we get above 0.85 (again out of 1.0), we've got a pretty good data collection system.

How does this fit into Lean/Six Sigma and management in general?

Whether we're leading a Lean/Six Sigma project or using data to guide us in everyday management, we want to know if our data is valid.  That's what a Measurement System Analysis (MSA) is all about, and a Kappa calculation is just one way of doing a MSA if our data is attribute/discrete data.

Of course, even if we prove that our data is valid, we should never rely on data alone.  We can surely use data to point us in the right direction to where problems are, but we should always go to the gemba and check out the gembutsu with our own eyes.  Never let data get in the way of facts.

Where can I learn more?

There's a guide I recommend, The Lean Six Sigma Pocket Toolbook by Michael L. George, et al.  Starting on pg. 100, you'll see a complete explanation for how to calculate Kappa.