Effective Level Four Evaluation - or Turning Kirkpatrick Upside Down

If you've been in training a while you've heard of evaluation and probably more specifically, Don Kirkpatrick's Four Levels of Evaluation. In shorthand they are:
1. Did they like it -- we use smile sheets at the end of the course.
2. Did they learn it? we test at the end (of couse without a pretest this is pretty much useless)!
3. Did they use it on the job? We followup and interview supervisors.
4. Did it make a difference? We identify metrics we want to see improved and look for improvement.

The problem is that the focus is too often on level 1 and 2 (85-95% of all companies) and too seldom (less than 5% of the time) on level four.

Here's my response to a blog that Don Clark does:

Don,
Several years ago I wrote an article for my Creative Training Techniques newsletter suggesting that we needed to turn Kirkpatrick upside down.
I served on ASTD's Board of Directors when Don was president in the early 80s and I count him as a friend.
Instead of eliminating his four levels, which have a lot of traction, my suggestion is to acknowledge them -- but start and focus on level four with your client by having a pain conversation.
Most of the difficulty with level four is that we try to do it academically. As training and performance profdessionals we are not doing research for a dissertation. We are trying to get results from the interventions we implement. The results that our clients are looking for.
The problem is that our clients often don't know what to ask for or to look for. Thus the need for a pain conversation.
What a pain conversation basically does is to help our client focus not on a solution, but on identifying the problem and the cost or consequence of the problem.
I'm running out of space, but I'll give a quick example:
A potential client called asking for a training program to improve their new teller orientation and training. Instead of providing that we asked, "Why?"
The response was that half the tellers quit after the training but before ever working as a teller.
Instead of simply responding with the requested training we helped the client dig a little deeper and we found:
1. The cost of hiring a teller and trainer was $15,000.2. Their were 40 tellers in a class 3X a year.So the cost (or pain) from this is $15K X 20 X 3= $900,0003. Transcaction errors among 240 tellers in 13 branches totaled 1200 per month. No one had ever quantified what this cost. The senior team came up with a cost of $50 per error X 1200 per month X 12 months = $720,000 per year.
We now have a $1.6 million dollar per year pain. We had asked the CEO to be part of the one day consult -- for the first hour -- he stayed all day.
We ended up with a different project than simply delivering a two day training program.
Within a month we discovered that over half the people starting the training never intended to be tellers. They were only there until they found something better. So we had a selection problem, not a training problem.
Within a month we uncovered the job factors of being a teller that caused the other half of the turnover during the training. We moved those things to the first week of the training so that the turnover accelerated -- they quit at the end of the first week of training -- not week five.
Withing six months we helped redesign the program using our participant-centered techniques to reduce training time from five weeks to three.
Within six months we identified what was causing the transaction errors on the job and put in place team strategies and accountability strategies that reduced trasaction errors from 1200 per month to 240 per month.
Within 18 months the bank was receiving a $1.2 million annual return on a one time investment of $350K.
The CEO was willing to spend $20k on training, but willingly spent $350k for solutions. Can we prove that the things I just described were what made the difference in the metrics? No. Not at an academic level. But from the viewpoint of the CEO, because we started with a pain conversation where he clearly identified the costs associated with the pain, not just the pain itself, he would say without reservation that it was our recommendations and help in implementing the recoemmendations that caused the dramatic shifts in the metrics. And in the world of business, that's what counts.
Kirkpatrick has always said of Level Four -- look for proof, not evidence. He just never said "how". The pain conversation is at least one "how to" that works well for us.
I hope your readers find this contribution to your blog useful.