Linstedt Daniel - Building a Scalable Data Warehouse with Data Vault 2.0

Publisher: Todd Green

Editorial Project Manager: Amy Invernizzi

Project Manager: Paul Prasad Chandramohan

Designer: Matthew Limbert

Morgan Kaufmann is an imprint of Elsevier

225 Wyman Street, Waltham, MA 02451, USA

Copyright 2016 Elsevier Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publishers permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

ISBN: 978-0-12-802510-9

I met Daniel Linstedt during a speech at Lockheed Martin in the early 1990s for the first time. By the time, he was an employee of the company, working for government projects. He approached me because he wanted my opinion about a concept that he had invented at the Department of Defense, in order to store large amounts of data. Back then, the term Big Data was not invented yet. But from what Daniel explained to me, the concept to deal with such huge amounts of data, was born.

Because back then, the end user had cried for give me my data!. But over time the end user became more sophisticated. The end user learned that it was not enough to get ones data. What a person needed was the RIGHT data. And then the sophisticated end user cried for give me my accurate and correct data!

The data warehouse represented the architectural solution to the issue of needing a single version of the truth. The primary reason for the existence of the data warehouse was the corporate need for integrity and believability of data. As such the data warehouse became the major architectural evolutionary leap beyond the early application systems.

But the data warehouse was not the end of architecture. Indeed, the data warehouse was only one stepping stone architecturally speaking in the progression of the evolution of architecture. It was Daniels idea that followed the data warehouse. In many ways the data warehouse set the stage for him.

Daniel used the term common foundational modeling architecture to describe a model based on three simple entities, focusing on business keys, their relationships and descriptive information for both. By doing so, the model closely followed the way business was using the data in the source systems. It allowed to source all kinds of data, regardless its structure, in a fully auditable manner. This was a core requirement of government agencies at the time. And due to Enron and a host of other corporate failures, Basel, and SOX compliance auditability was pushed to the forefront of the industry.

Not only that, the model was able to evolve on changing data structures. It was also easy to extend by adding more and more source systems. Daniel later called it the Data Vault Model and it was groundbreaking.

The Data Vault became the next architectural extension of the data warehouse. But the Data Vault concept like all evolutions continued to evolve. He asked me what to do about it and, as a professional author, I gave him the advice to publish the heck out of it. But Daniel decided to take it to the long run. Over multiple years, he improved the Data Vault and evolved it into Data Vault 2.0. Today, this System of Business Intelligence includes not only a more sophisticated model, but an agile methodology, a reference architecture for enterprise data warehouse systems, and best practices for implementation.

The Data Vault 2.0 System of Business Intelligence is ground-breaking, again. It incorporates concepts from massively parallel architectures, Big Data, real-time and unstructured data. And after all the time, Im glad that he followed my advice and has started to publish more on the topic.

This book represents that latest, most current step in the larger evolution of the Data Vault that has been occurring. This book had been carefully and thoughtfully prepared by leaders in the thought and implementation of the Data Vault.