Patricia Pulliam Phillips - Real World Training Evaluation: Navigating Common Constraints for Exceptional Results

2016 ASTD DBA the Association for Talent Development (ATD)

All rights reserved. Printed in the United States of America.

19 18 17 16 1 2 3 4 5

No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, please go to www.copyright.com, or contact Copyright Clearance Center (CCC), 222 Rosewood Drive, Danvers, MA 01923 (telephone: 978.750.8400; fax: 978.646.8600).

ATD Press is an internationally renowned source of insightful and practical information on talent development, training, and professional development.

ATD Press

1640 King Street

Alexandria, VA 22314 USA

Ordering information: Books published by ATD Press can be purchased by visiting ATDs website at www.td.org/books or by calling 800.628.2783 or 703.683.8100.

Library of Congress Control Number: 2015953307

ISBN-10: 1-56286-907-8

ISBN-13: 978-1-56286-907-6

e-ISBN: 978-1-60728-218-1

ATD Press Editorial Staff

Director: Kristine Luecker

Manager: Christian Green

Community of Practice Manager, Learning & Development: Amanda Smith

Associate Editor: Melissa Jones

Text Design: Iris Sanchez and Maggie Hyde

Cover Design: Iris Sanchez

Printed by Data Reproductions Corporation, Auburn Hills, MI

In the early 1970s, the real world of training included minimum evaluation. Donald Kirkpatrick completed his doctoral dissertation in the 1950s, bringing a concept of four steps to training evaluation to the field. While an interesting concept, it was missing a model and set of standards to support replication and scalability over time. At the time, organizations collected data at the reaction and learning levels, with minimal, if any, data collection occurring at the application and impact levels. Return on investment (ROI) was not even a twinkle in the eyes of learning professionals.

Then in 1972 Jack Phillips brought the real world of business to the training profession. In response to a senior executives request to demonstrate the financial ROI of a cooperative education program, he built upon Kirkpatricks four steps to create what are now known as the five levels of evaluation. Additionally, he developed a model and set of standards, or guiding principles, to ensure credibility and reliability in his approach. This triadthe five-level framework, 10-step process model, and 12 guiding principlesis known as the ROI Methodology, which is currently the most applied and documented approach to evaluate talent development programs and a variety of other programs in other disciplines. This process provides a system for talent development professionals to ensure their programs drive the results stakeholders need to make decisions. In todays world, the need for evaluation of talent development programs has never been so real.

For the past three years, The Conference Boards CEO Challenge, an annual survey of chief executive officers at large organizations around the world, has indicated that the top challenge CEOs face is human capital. Their continued cry for talent, coupled with the July/August 2015 Harvard Business Review cover story Its Time to Blow Up HR, indicates something isnt working. CEOs need human capital to help them drive operational excellence, customer relationships, innovation, and sustainability. According to the CEO Challenge 2015, training and development and improving leadership development programs are among the top strategies to address this need.

Unfortunately, demonstrating how training and development, leadership development, and other talent development initiatives drive operational excellence, customer relationships, innovation, and sustainability has not been top-of-mind for many learning organizations. While there are hundreds of studies describing the impact and ROI these programs contribute to the business, the demand outweighs the supply. In 2009 and 2010, we partnered with ATD to ask CEOs about data they receive for learning investments: What did they want to receive and how important were the different types of measures? They told us the top two measures were impact and ROI, respectively. But they also told us that they werent receiving these measures of results. This gap is tellingthe focus of evaluating talent development remains, in large part, on the inputs and lower levels of data. The good news is that things are changing, thanks to increased application of the ROI Methodology, ongoing research that calls for more focus on impact and ROI data, and louder cries from executives.

This change is demonstrated in the results of Chief Learning Officers Business Intelligence Board 2015 Measurement and Metrics survey. According to this report, 35.6 percent of the 335 respondents use business impact data to describe the results of training on the broader enterprise; 21.6 percent use ROI for the same purpose. In terms of planning, 22.6 percent plan to implement ROI in the next 12 months and 9.7 percent plan to implement ROI within 12 to 24 months of the study. In addition, 17.3 percent plan to implement it with no particular timeframe in mind. This suggests that 71.2 percent of respondents are either using or plan to use ROI data as part of their measurement mix.

You may be thinking, Not another book on evaluation. On the one hand, we agree with you. But, the conversation about measurement and evaluation continuesparticularly in this era of all things analytics.

Think about it. Human capital represents the top challenge for CEOs. Investment in human capital runs the gamut. McKinsey & Company and The Conference Boards The State of Human Capital 2012: False Summit reported that organizations still have far to go when it comes to human capital, and that the common thread among all things human capital is analytics. Analytics requires measurement and evaluation; it requires collecting, analyzing, reporting, and using data. While much of the analytics conversation is at the macrolevel, where organizations attempt to claim success based on high-level statistical models describing the connection between overall investment in people and overall organization performance, it is at the program-investment level where the rubber meets the road. It is through implementation of training and development programs, performance management systems, diversity and inclusion initiatives, engagement projects, and leadership development programs that the results occur. By demonstrating results at the program level, it is possible to show how that micro-investment leads to the macro-results.

Measurement and evaluation, including impact and ROI, are the foundation for linking the human capital investment to operational excellence, customer relationships, innovation, and sustainability. They remain hot topics, and will remain so for years to come. The question is whether or not wethose of us perpetuating the conversationcan keep elevating the conversation so that organizations who have trekked beyond base camp can continue their climb toward the measurement and evaluation summit.