Megan Torrance - Data & Analytics for Instructional Designers

2023 ASTD DBA the Association for Talent Development (ATD)

All rights reserved. Printed in the United States of America.

26 25 24 23 1 2 3 4 5

No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, information storage and retrieval systems, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law. For permission requests, please go to copyright.com, or contact Copyright Clearance Center (CCC), 222 Rosewood Drive, Danvers, MA 01923 (telephone: 978.750.8400; fax: 978.646.8600).

ATD Press is an internationally renowned source of insightful and practical information on talent development, training, and professional development.

ATD Press

1640 King Street

Alexandria, VA 22314 USA

Ordering information: Books published by ATD Press can be purchased by visiting ATDs website at td.org/books or by calling 800.628.2783 or 703.683.8100.

Library of Congress Control Number: 2022949753

ISBN-10: 1953946445

ISBN-13: 9781953946447

e-ISBN: 9781953946454

ATD Press Editorial Staff

Director: Sarah Halgas

Manager: Melissa Jones

Content Manager, L&D: Jes Thompson

Developmental Editor: Jack Harlow

Production Editor: Hannah Sternberg

Text and Cover Design: Shirley E.M. Raybuck

Printed by BR Printer, San Jose, CA

Ive been working in L&D since 2002, when I helped a large healthcare organization implement their first learning management system (LMS). It was the software companys first LMS, and mine too. These were the early days of SCORM (sharable content object reference model). The e-learning industry was about to take off as the advent of rapid authoring tools and LMSs began to democratize access to scale.

In 2012, I learned about Project Tin Can, which would create the Experience API, or xAPI. Our team at TorranceLearning had been seeking a learning and performance environment that offered a richer and more varied learning experience, and a correspondingly interesting data set. We were stretching our technical muscles. Our instructional designers were grappling with the new grammar of reporting data in a repeatable but not-yet-standardized environment. We were asking our LMS team: Shouldnt this do xAPI, too? (The answer: Why, yes, yes it should.)

In 2014, we launched the Ann Arbor Hands-On Museums Digitally Enhanced Exhibit Program (DEEP). Student groups on field trips would use beacons to identify themselves to the networked tablets placed around the museum. They engaged seamlessly with interactions and questions that would be recorded by exhibit and curriculum standard. At the end of the visit, teachers would receive a stack of reports about their students activities and engagement, and each student received a personalized one-page report detailing their field trip to the museum. This was exciting stuff. We often kept coming back to the question: What can we do with all this data? How can we take advantage of it? What insights might lie in there?

In 2015, our team took on the duty of hosting the Advanced Distributed Learning (ADL) groups cohort model for introducing innovators to xAPI. We started our first 12-week xAPI Learning Cohort with a group of 35 invited designers and developers in the fall of 2015, and ran two a year for the next seven years. Cohorts routinely exceeded 600 members each semester and the projects that teams took on ranged from e-learning to gaming to chat bots to virtual classroom to Alexa and everything in between. As of this writing, more than 5,000 L&D professionals have participated in the xAPI Learning Cohort.

As cohort members and organizations adopted xAPI and other data-rich learning experiences, we kept coming back to the question, What can we do with all of this data? Thats what this book sets out to help you answer.

In the early days of the COVID-19 pandemic, the L&D team the UK arm of PriceWaterhouseCoopers used their analysis of search requests on the organizations learning experience platform to identify the needs being faced by their staff and managers. This allowed the L&D team to respond very quickly, almost at a week-by-week pace, to these emerging needs.

At LeMoyne Institute in upstate New York, the learning design team used detailed data from a learning experience to fine-tune the screen design for an adaptive e-learning curriculum. By simply changing the layout of the screen, they could improve relevant performance in measurable ways.

QuantHub, a learning experience platform focused on data science, uses data to personalize learning across a competency map used by major organizations to upskill their professionals.

At Trane technologies, learner and manager feedback are combined with employee engagement survey results to prove the positive impact of their leadership courses. This data is used to obtain additional budget to continue running the program, as well as to attract new learners to the experience.

And as interesting as these quick case studies are, there are countless organizations using data and analytics in similar ways to identify learning needs, hone the design of their learning experiences, personalize learning in new ways, support decisions, and evaluate the impact of learning. Well hear from several of them in this book, at the end of each chapter.

This book tackles an unaddressed need in the market for workplace learning and talent development.

First, there are lots of books, articles, courses, and academic degrees in data and analytics. However, I find they tend to be focused on the marketing, sales, or operational aspects of a business, where the data is rich and the metrics are commonplace. Its not very often that I see an analytics case study that addresses the kinds of data we are using in L&D.

Second, with K12 and higher education maturing in their use of learning management systems, and the popularity of the MOOC (massive open online course), the academic field is investing in student analytics. There is much to be learned from our academic colleagues for sure, but their analytics work doesnt fully account for the workplace setting.

Third, there are handfuls of books about learning measurement in the corporate space, going into the familiar evaluation levels developed by Donald Kirkpatrick, Raymond Katzell, and Jack Phillips, and beyond them to address the culture and practice of regular data gathering, analysis, and reporting. In fact, if this is your interest, I strongly recommend Measurement Demystified by David Vance and Peggy Parskey (2021) and Learning Analytics by John Mattox, Peggy Parskey, and Cristina Hall (2020).

Missing across these resources is a focus on the unique data that is attainable in the corporate learning space at a granular level, as well as specific direction for instructional design teams about how to generate this data to feed the downstream uses.

Why? Im sure there are several reasons for this. Chief among them is that in L&D we tend not to have as much data at our fingertips as other functions in the business, and therefore tend not to use data to drive our decisions. In most organizations, finance, sales, and operations all have very granular data available within a few clicks to drive their decision making. In L&D we have training completion data: Did learners complete the training? When? How long did it take? What are the test scores? Did they like it? Are they motivated to apply it?