Richard Swinbank - Azure Data Factory by Example: Practical Implementation for Data Engineers

Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the books product page, located at www.apress.com/9781484270288. For more detailed information, please visit http://www.apress.com/source-code.

This Apress imprint is published by the registered company APress Media, LLC part of Springer Nature.

The registered company address is: 1 New York Plaza, New York, NY 10004, U.S.A.

Azure Data Factory (ADF) is Microsofts cloud-based ETL service for scale-out serverless data movement, integration, and transformation. The earliest version of the service went into public preview in 2014 and was superseded by version 2 in 2018. ADF V2 contains so many improvements over V1 that it is all but a different product, and it is on ADF V2 that this book is exclusively focused.

From the outset, a major strength of ADF has been its ability to interface with many types of data source and to orchestrate data movement between them. Data transformation was at first delegated to external compute services such as HDInsight or Stream Analytics, but with the introduction of Mapping Data Flows in 2019 (now simply Data Flows), it became possible to implement advanced data transformation activities natively in ADF.

ADF can interact with 100 or more types of external service. The majority of these are data storage services databases, file systems, and so on but the list of supported compute environments has also grown over time and now includes Databricks, Azure Functions, and Azure Machine Learning, among others. The object of this book is not to give you the grand tour of all of these services, each of which has its own complexities and many of which you may never use. Instead, it focuses on the rich capabilities that ADF offers to integrate data from these many sources and to transform it natively.

Azure Data Factory is evolving and growing rapidly, with new features emerging with every month that passes. Inevitably, you will find places in which the ADF User Experience (ADF UX) differs from the screenshots and descriptions presented here, but the core concepts remain the same. The conceptual understanding that you gain from this book will enable you confidently to expand your knowledge of ADF, in step with the development of the service.

Readers of the book will be aware of Azure Synapse pipelines, a serverless ETL service parallel to ADF inside Azure Synapse Analytics. Although this book makes no explicit reference to Synapse pipelines, many of the concepts and tools are immediately transferable. At the time of writing, Synapse pipelines are some distance from achieving feature parity with ADF for the time being, readers using Synapse pipelines should be prepared for the absence of certain ADF features.

The book is designed with the working data engineer in mind. It assumes no prior knowledge of Azure Data Factory so is suited to both new data engineers and seasoned professionals new to the ADF service. A basic working knowledge of T-SQL is expected.

If you have a background in SQL Server Integration Services (SSIS), you will find that ADF contains many familiar concepts. The For SSIS developers notes inserted at various points in the text are to help you leverage your existing knowledge or to indicate where you should be aware of differences from SSIS.

The book uses a series of tutorials to get you using ADF right away, introducing and reinforcing concepts naturally as you encounter them. To undertake exercises, you will need access to an Azure subscription and a web browser supported by the ADF UX browsers supported currently are Microsoft Edge and Google Chrome. Choose a subscription in which you have sufficient permissions to create and manage the various Azure resources you will be using. Chapter .

Work through the chapters in order, as later chapters rely on both knowledge and ADF resources developed in earlier chapters. When directed to give a resource a specific name, do so, because that name may later be used to refer back to the resource. References to labels in user interface components, for example field names or page titles, are given in italics. Input values, for example for text box input or radio button selection, are given in quotes when you are asked to enter a value given in quotes, the quotes should not be included unless you are directed to do so.

While this book is about one specific service Azure Data Factory it is the product of years of experience working as a data engineer. I am enormously grateful to the many colleagues, past and present, from whom I continue to learn every day. Im indebted to the wider Microsoft data platform community, a group of engaged, generous people who are unstinting in their advice and support for others working in this space.

I want to thank my technical reviewer, Paul Andrew, for innumerable conversations which have made the book many times better than it could otherwise have been. Paul is a real expert in this technology, and Im very fortunate to have benefited from his advice. I must thank Simon Swinbank and Liz Bell for their invaluable input in cleaning up and clarifying the text. Thanks also to the editorial team at Apress, Jonathan Gennick, Jill Balzano, and Laura Berendson, without whom this book would not have been possible.

Finally, to Catherine, who has been nothing but supportive and encouraging throughout the length of this project I thank you from the bottom of my heart.