Ken W. Collier - Agile Analytics: A Value-Driven Approach to Business Intelligence and Data Warehousing

A Value-Driven Approach to Business Intelligence and Data Warehousing

Ken Collier

Upper Saddle River, NJ Boston Indianapolis San Francisco
New York Toronto Montreal London Munich Paris Madrid
Capetown Sydney Tokyo Singapore Mexico City

This book does a great job of explaining why and how you would implement Agile Analytics in the real world. Ken has many lessons learned from actually implementing and refining this approach. Business Intelligence is definitely an area that can benefit from this type of discipline.

Dale Zinkgraf, Sr. Business Intelligence Architect

One remarkable aspect of Agile Analytics is the breadth of coveragefrom product and backlog management to Agile project management techniques, from self-organizing teams to evolutionary design practices, from automated testing to build management and continuous integration. Even if you are not on an analytics project, Kens treatment of this broad range of topics related to products with a substantial data-oriented flavor will be useful for and beyond the analytics community.

Jim Highsmith, Executive Consultant, ThoughtWorks, Inc., and author of Agile Project Management

Agile methods have transformed software development, and now its time to transform the analytics space. Agile Analytics provides the knowledge needed to make the transformation to Agile methods in delivering your next analytics projects.

Pramod Sadalage, coauthor of Refactoring Databases: Evolutionary Database Design

This book captures the fundamental strategies for successful business intelligence/analytics projects for the coming decade. Ken Collier has raised the bar for analytics practitionersare you up to the challenge?

Scott Ambler, Chief Methodologist for Agile and Lean, IBM Rational Founder, Agile Data Method

A sweeping presentation of the fundamentals that will empower teams to deliver high-quality, high-value, working business intelligence systems far more quickly and cost effectively than traditional software development methods.

Ralph Hughes, author of Agile Data Warehousing

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals.

The author and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales
(800) 382-3419

For sales outside the United States please contact:

International Sales

Visit us on the Web: informit.com/aw

Library of Congress Cataloging-in-Publication Data

Collier, Ken, 1960
Agile analytics : a value-driven approach to business intelligence and
data warehousing / Ken Collier.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-321-50481-4 (pbk. : alk. paper)
1. Business intelligenceData processing. 2. Business
intelligenceComputer programs. 3. Data warehousing. 4. Agile
software development. 5. Management information systems. I. Title.
HD38.7.C645 2012
658.472dc23
2011019825

Copyright 2012 Pearson Education, Inc.

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to:

Pearson Education, Inc.
Rights and Contracts Department
501 Boylston Street, Suite 900
Boston, MA 02116
Fax: (617) 671-3447
ISBN-13: 978-0-321-50481-4
ISBN-10: 0-321-50481-X

Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana.

First printing, July 2011

This book is dedicated to my wife and best friend, Beth, who never once asked, How come its taking you so long to finish that darn book?

I was introduced to Ken Collier through a mutual friend about seven years ago. We started meeting for coffee (a two-person Agile group in Flagstaff, Arizona) every week or so to talk about software development, a sprinkling of Agile here and there, skiing, mountain biking, and Kens analytics projects. Early on, as Ken talked about a project that was faltering and I talked about Agile, he decided to try out Agile on his next project. As he quipped, It couldnt be worse!

Over the years Ive heard every reason imaginable why Agile wont work in my company because we are different. Ken never had that attitude and from the beginning kept trying to figure out not if Agile would work on business intelligence and data warehousing projects, but how it would work. Ken saw each impediment as an opportunity to figure out an Agile way to overcome it. From developing user stories that traversed the entire analytics software stack, to figuring out how to do continuous integration in that same diverse stack, Ken has always been Agile, just as he was learning to do Agile. Today, Ken champions the cause of being Agile and not just doing Agile.

Over subsequent analytics projects, one that ran for over three years, delivering releases every quarter, Ken took the fundamental Agile management and development practices and came up with innovative ways to apply them. Business intelligence and data warehousing developers have been reluctant to embrace Agile (although that is changing) in part because it wasnt clear how to apply Agile to these large, data-centric projects. However, analytics projects suffered from the same problems as more typical IT projectsthey took too long, cost too much, and didnt satisfy their customers. In our current turbulent business era these kinds of results are no longer acceptable.

One remarkable aspect of Agile Analytics is the breadth of coveragefrom product and backlog management, to Agile project management techniques, to self-organizing teams, to evolutionary design practices, to automated testing, to build management and continuous integration. Even if you are not on an analytics project, Kens treatment of this broad range of topics related to products with a substantial data-oriented flavor will be useful for and beyond the analytics community.

In each subject area he has taken the basic Agile practices and customized them to analytics projects. For example, many BI and data warehouse teams are far behind their software development counterparts in configuration management. With execution code in Java, Ruby, and other languages, stored procedures, SQL, and tool-specific code in specialized tools, analytics teams often have poor code management practices. Ken spends several chapters on reviewing techniques that software developers have been using and showing how those techniques can be adapted to an analytics environment. Ken often asks analytics teams, If your servers went down hard today, how long would it take you to rebuild? The responses he typically receives vary from a few weeks to never! The automation of the build, integration, and test process is foreign to many analytics teams, so Ken spends a chapter each on version control and build automation, showing how to build a fast-paced continuous integration environment.