Effectively Evaluating Impact – Where to start?

Evaluating the impact of training is essential to ensure it goes beyond mere completion rates and truly drives meaningful change in the workplace. By reversing the Kirkpatrick model and focusing on the desired outcomes from the outset, organisations can align training initiatives with their strategic goals, promoting desired behaviours that lead to measurable results.

Most organisations have at least one problem that needs solving. Whether it is staff retention, safety issues, skill gaps, customer satisfaction, sales shortages or poor job performance to name just a few, the overwhelmingly common approach is to throw training at this problem and hope it improves.

This may be effective in some cases, but the majority find themselves looking for answers as to why training has not been effective, ending up where they had started.

To maintain effectiveness, training needs to focus on impact

One of the commonly used metrics in workplace learning is completion. This indicates that a learner has commenced through a digital learning offering, has possibly successfully answered quiz questions and any other pre-requisite content set forth by the organisation.

The problem with relying on a completion as an outcome, is that there is no real way of knowing the level of understanding and the learner’s ability to apply what they have learned into the workplace. Understanding if there has been an improvement in their performance, due to the training is often anecdotal, and based on very broad metrics.

Evaluation methods go some way to addressing this

There are various methods for gathering feedback and evaluating effectiveness. The Kirkpatrick Evaluation method is a commonly used example, with several iterations and updates. Kirkpatrick provides a uniform and reliable set of rules and approaches that help teams structure their evaluations and ultimately report on their outcomes.

The Kirkpatrick method consists of 4 levels of evaluations which utilise observation, interviews, surveys, and various measurement tool across four levels of evaluation.

The 4 levels of Kirkpatrick

Reaction
The first level measures participants’ immediate reactions to training. In facilitator-led sessions, trainers often use evaluation forms or satisfaction surveys. Collecting and analyzing this data can feel manual and time-consuming unless you centralize it in one location.

Learning
Trainers measure what participants have learned by aligning outcomes with assessments. Common methods include online quizzes in different formats or written tests. New technologies now provide more effective and varied options for assessment. In addition to knowledge checks, it is important to measure confidence, motivation to change, and readiness to apply the learning.

Behaviour
This level looks at whether participants apply what they learned and change their behaviour. Measuring behaviour often requires weeks or months of observation. Organisational culture, peer and manager influence, and unclear processes can affect behaviour change. To succeed, trainers need to create drivers that support employees in applying desirable behaviours.

Results
This level identifies the outcomes training produces for the organisation. Trainers can measure results through observations, interviews, and reviews of key metrics. A common challenge arises when teams search for results retrospectively, hoping to connect existing organisational outcomes to training. Instead, they should define the results they want in advance and design training to achieve them.

Kirkpatrick is a good view of the past, it may miss on immediacy and dynamism

The Kirkpatrick method has been compared to a black box. It is good for useful information, but that is only available after the plane has crashed!

While Kirkpatrick provides a good basis to measure outcomes, it is often used in retrospect, measuring what has occurred in the past. In today’s work environment and with the technology available to us, waiting until the evaluations return in order to gain insight into how effective it has been puts those who may need assistance at risk.

Reversing Kirkpatrick helps to focus on the outcomes and people

Reversing the order of the Kirkpatrick levels of evaluation gives structure and guidance to a learning and development strategy. This assists in creating impact based learning.

Begin with a view of the results you require

Understanding what tangible metrics that will define success is the first place to start. This should reflect organisational goals such as solving and preventing problems from occurring, continuous improvement, maintaining profitability or any multitude of others. They may be hard metrics such as sales, or more people focused like turnover rates, the key thing is ensuring the goals translate to something meaningful to the organisation and have tangible metrics associated with them.

An example of this could be an organisation that has a high turnover of staff. The metric being attrition rate. They know they have a problem, so looking at the metrics that are available to them, such as employee satisfaction, performance appraisals or career progression goals, can help to start setting goals to improve their attrition rate.

Focus on the behaviours will help you reach your goals

It’s through changes in behaviour that you achieve your results. Once you know your goals and the metrics you hope to achieve, you can outline the behaviours that you hope to see in your people that will lead to achieving your goals.

To continue the staff attrition example, you would hope to see behaviours consistent with engaged and motivated employees. This may be nailing performance appraisals, being accountable, working collaboratively, providing and openly receiving feedback, completing and seeking any new learning and development opportunities and proactively seeking out ways to go above and beyond.

The key is to identify these desirable behaviours and ensure there are adequate methods in place to capture and measure these.

Combine learning and reactions to enable dynamic experiences

Level 1 and 2 of Kirkpatrick are the participants’ reactions to the learning and if they learned anything. When reversing Kirkpatrick and focusing on Impact and Results, we look at what behaviours and results we established earlier to help determine the best approach to learning experiences.

In the example we used earlier of wanting to encourage engaged employees with positive behaviours that result in reducing attrition rates. Provide a training opportunity such as an onboarding program, induction or career progression goals to suit the goals.

For example:

Onboarding – Learning experience aimed at teaching what accountability looks like with scenarios and possible social interaction with other new starters. After completing exercises etc, employees can provide feedback in the form of sentiment and comfort with this behaviour. Use that data to improve the experience, intervene when employees show at-risk tendencies, and compare it with future job performance to find correlations with training.

Evaluating the impact of training is essential to data goes beyond completion rates and truly drives meaningful change in the workplace. By reversing the Kirkpatrick model and focusing on the desired outcomes from the outset, organisations can align training initiatives with their strategic goals. This proactive approach not only improves the effectiveness of training but also helps create a more engaged, capable workforce that can drive the organisation forward, ensuring long-term success and growth.