Earlier this month, we discussed strategies to build an effective learning program. Using specific solutions, such as initial assessments to establish a training need, understanding the business needs and goals, and measurement mapping, have proven to be essential in the process of designing a learning program.

These strategies can be more effective by integrating the use of technology. Technology plays an increasing role in how useful a training event is to learners and its impact on achieving business goals. Proper course tracking and data sharing allows organizations to develop best practices for measuring the impact of a learning program. However, management teams fall short when it comes to adapting to these new processes.

With these process gaps and management challenges in mind, there are ways for L&D professionals to:

1. Use technology to ensure your learning program is helping your organization meet business goals.
2. Identify the proper tools for measuring the success of your learning program.

Are You Maximizing Your Data Captures?

You need to leverage the knowledge of what is not successful and why. Establish which metrics are essential to the success of your program and how to effectively utilize technology to facilitate data analyses. Be sure to include an evaluation strategy and collect assessment data to make informed recommendations.

This is where the Kirkpatrick measurement model comes into play. Developed in 1955 as part of Donald Kirkpatrick’s Ph.D. dissertation, it’s still one of the most common methods for evaluating a learning solution’s effectiveness. Its composition includes four levels, typically followed as a linear process:

1. Reaction — The degree to which participants find the training favorable, engaging, and relevant to their job.
2. Learning — The degree to which participants acquire the intended knowledge, skill, attitude, confidence, and commitment based on their participation in the training.
3. Behavior — The degree to which participants apply what they learned during training when they’re back on the job.
4. Results — The degree to which targeted outcomes occur as a result of the training and the support and accountability package.

Assessing the Kirkpatrick Learning Measurement Model

Although the Kirkpatrick Model has been widely used since its introduction 60 years ago, 91% of companies were only using Kirkpatrick at Levels 1 and 2. That means they’re only measuring enrollments, completions, learner satisfaction, and test scores. Neglecting to measure job impact (Level 3) and organizational impact (Level 4) indicates this popular method’s misuse.

But why?

Historically speaking, tools and systems haven’t made accessing and correlating data any easier to measure the success of learning. For example, respondents to Udemy's 2018 'State of the ROI of Learning Report' stated they do not capture data for levels three and four because they are unsure of how to do so.

So, how can we continue to work toward this goal and ensure we remain strategic business partners? It all comes down to finding the proper combination of strategies and technology to help you optimize learning results that increase performance and impact business outcomes.

Recommended related reading: 'The 13 Must-Have Features of a Learning Management

Capturing Data with Technology

Data and information aren’t interchangeable; they’re sequential. With the use of a Learning Management System (LMS), SCORM interactions, xAPI, completion reports and feedback from co-workers, mentors, or managers, there can be an overwhelming amount of data captured for our learning events. Making the delineation between data captured without a specific purpose in mind and data that can be usefully converted into meaningful information will help make the task less daunting.

Many data scientists like to use the phrase ‘Make your data talk.’ Once data has been converted into useful information, it must be applied to specific work or performance as an intervention. At this point, the data has become knowledge, and therefore, can be acted on to have a more effective impact in adding value to an organization.”

LMS users also have formal tools available to them such as course evaluations, sometimes known as smile sheets. If these tools are currently used to capture Level 1 and 2 data, you can re-write questions or add new ones to set the foundation for capturing Levels 3 and 4. A Learning Experience Platform (LXP) also lets you capitalize on important data generated by the xAPI transactions through the Learning Record Store (LRS).

For example, the yes or no question “Was this material valuable to you?” can be re-written to solicit valuable feedback from your learner. “Identify two ways you can implement this new information into your environment” is a request that allows us to analyze two things:

• Does the learning event allow an employee to use the information in their role? (Level 3)
• Did the event elicit results? (Level 4)

More from the blog: '4 Ways Learning Technologies Can Boost Learner Engagement'

Using Technology to Measure the Success of Learning

It’s important for managers to understand how to use formal and informal tools to capture data. The biggest challenge is using this data to develop an actionable plan that gives way to measuring the learning program’s efficiency. One of the best ways to do this is by aggregating and categorizing the data until it ‘tells a story.’

There are several resources available for L&D professionals to measure beyond reaction to impact. A few recommendations of resources include:

• Standard completions through xAPI, which is considered the experience. For example, Sam completed compliance training on January 3rd at 12:17pm EST instead of Sam completed a task on January 3rd. The variables from our first example can be fed to analytics software and analyzed to track the impact.
• Data Scientist roles have only been in the industry for the last 10 years. These individuals interpret the results of our learning events and explain it in terms we can take action on.
• The New World Kirkpatrick Model was introduced in 2016 and is a modern application of its original model. The book, “Four Levels of Training Evaluation” by James D. Kirkpatrick and Wendy Kayser Kirkpatrick, is based on this modern tool. It provides examples on how to rewrite Level 1 and 2 questions in a more probing manner to solicit valuable feedback versus the yes or no queries.

In addition to these resources, video scenarios where learners can comment on applicable solutions have proven to encourage a more interactive learning experience and help L&D professionals to identify gaps. On-demand videos, such as ‘how-to’ guides are especially successful in this arena.

Final Thoughts on Measuring the Success of Learning

As L&D professionals, it’s our responsibility to ensure we provide an effective learning program. With the use of formal and informal strategies, as well as a proper understanding of how to use the technology available, we can begin to measure the success of learning.

It’s important to remember that our work doesn’t end after the capture of data. We have to find creative ways to ‘tell a story’ with the data we’ve analyzed and converted into information. In doing so, we can develop a deeper understanding of our learners and how our program impacts their performance.

The technologies and tools to measure the efficiency of your learning program come from the webinar ‘Strategy + Technology: A Winning Combination for Effective Learning.’ Click here to watch the full webinar recording.