Simon Trewin - The DataOps Revolution: Delivering the Data-Driven Enterprise

Delivering the Data-Driven Enterprise

Delivering the Data-Driven Enterprise

Simon Trewin

First Edition published 2022
by CRC Press
6000 Broken Sound Parkway NW, Suite 300, Boca Raton, FL 33487-2742

and by CRC Press
2 Park Square, Milton Park, Abingdon, Oxon, OX14 4RN

2022 Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, LLC

The right of Simon Trewin to be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

Reasonable efforts have been made to publish reliable data and information, but the author and publisher cannot assume responsibility for the validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying, microfilming, and recording, or in any information storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, access

Trademark notice: Product or corporate names may be trademarks or registered trademarks and are used only for identification and explanation without intent to infringe.

ISBN: 978-1-032-10893-3 (hbk)
ISBN: 978-1-032-06296-9 (pbk)
ISBN: 978-1-003-21979-8 (ebk)

Excel is a registered trademark by Microsoft Corporation in Redmond, WA. Tableau is a registered trademark by Tableau Software, LLC, in Seattle, WA. Qlik is a registered trademark by QlikTech International AB in Lund. Spark is a registered trademark by Motorcity Systems LLC in Romulus, MI. Data Vault is a registered trademark by Data Defenders LLC in Chicago, IL.

• Contents

I would like to thank many people for the opportunity to write this book. First and foremost, my family, Sarah, Joshua, and Toby, who have supported me on the writers journey.

I got into data systems early in my career, and I have never looked back. During that time, I have picked up tips, tools, knowledge, and experience from many people. This is divided into technical, architectural, personal, cultural, and organisational skills. Delivering data solutions requires skills in all these areas, and it would be hard to mention everyone on that journey. If you were part of that path, then thank you for working with me and providing the guidance and advice.

I would like to acknowledge the many influences I have been given over the years through reading numerous books. The DataOps Revolution contains many techniques that were first described by others. It has been my privilege to learn from these experts in their fields, applying them to my experiences and writing them into this story and the design of the Six IMPACT Pillars.

I started my journey in data at University, winning the Addison-Wesley Prize for computer science and receiving a copy of The Relational Model for Database Management by Edgar F. Codd.

I progressed to building data systems for warehousing financial data in the big banks, and a big influence was The Data Warehouse Toolkit by Ralph Kimball and Margy Ross.

I had huge success rolling out Kimball warehouses for years but needed a better way to store the data in the pipeline, leading up to the analytical schemas which brought me to Building a Scalable Data Warehouse with Data Vault 2.0 by Daniel Linstedt and Michael Olschimke.

_______________

Building automation into the pipeline required me to learn from the DataOps pipeline automation gurus in The Data Ops Cookbook by Christopher Bergh, Gil Benghiat, and Eran Strod.

The style of this book aligns to the story method of writing, and I first really enjoyed another Addison-Wesley Prize book, The Mythical Man-Month by Frederic P. Brooks, Jr.

I would like to acknowledge the many influences that I have had to help to describe what are some very complex ideas in everyday language and imagery. Simon Sineks Infinite Game is an excellent explanation of data processes that are infinite games played out by finite budget holders.

Finally, I would like to thank Andy Harrington for his excellent speakers academy and his Tri-Summit Story Telling System and Coaching.

I hope that this style of book really gets you excited about implementing your data projects in a way that brings the joy back to the process of building data and analytics systems.

_______________

Simon Trewin is an IT professional and entrepreneur who has had the privilege of working with many leading organisations in delivering data solutions for over 25 years. He has a passion for data and analytics that has led him to structuring the Six IMPACT Pillars of DataOps, and he is the founder of the DataOps Thinktank on Linkedln with 600+ members.

Simon is a regular speaker and presenter at data conferences and has published many articles on LinkedIn and social media. Key successes in his career have come from helping organisations to accelerate their data delivery using a combination of tried and trusted solutions and new thinking to enable them to meet challenging deadlines and overcome complex problems.

Simon has used DataOps techniques for the past 15 years. In that time, he has lead projects including:

• Successfully delivering data and analytics for a trading business through the financial crisis, enabling it to remain profitable throughout.
• Turning around a regulatory crisis at a large investment bank to deliver a solution in eight weeks that the bank in question had struggled with for six months.
• Building a team and delivering quality solutions to a large UK retail bank that satisfied a regulation within six months that was not being satisfied by a huge IT program.

The tools and techniques that he has learnt to achieve these outcomes are described in this book, and they are things that will improve how the reader implements successful projects with data and analytics. He has written this book to help people to make a step change in their capabilities.

Simon is co-founder and CEO of Kinaesis, which is the publisher of many articles in relation to DataOps. Kinaesis work with clients within financial services to leverage investment in data solutions and generate real value. Kinaesis have consistently achieved rapid success at a lower cost and risk for their clients in this complex and dynamic arena.

In my experience working on, leading, and delivering many data and analytics solutions, it is building collaboration that has been the key to creating lasting, utilised, and innovative solutions.