Michael W Lucas - FreeBSD Mastery: ZFS

FreeBSD Mastery: ZFS

Copyright 2015 by Michael W Lucas and Allan Jude.

All rights reserved.

Authors: Michael W Lucas and Allan Jude

BSD Daemon Copyright 1988 by Marshall Kirk McKusick. All rights reserved.

Copyediting: Lindy Lou Losh

Cover art: Beastie-cycling , illustration copyright 2015 Eddie Sharam, after Bicycling , 1887, by Hy Sandham.

ISBN-13: 978-0692452356

ISBN-10: 0692452354

All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, cuneiform, or by any information storage or retrieval system, without the prior written permission of the copyright holder and the publisher. For information on book distribution, translations, or other rights, please contact Tilted Windmill Press (accounts@tiltedwindmillpress.com)

Product and company names mentioned herein might be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

The information in this book is provided on an As Is basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor Tilted Windmill Press shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

Tilted Windmill Press

https://www.tiltedwindmillpress.com

The authors would like to thank all the people who have helped us with this book in one way or another. That includes a whole bunch of people on the FreeBSD mailing lists, lots of folks on social media, and every customer whos ever damaged their filesystem.

Wed also like to thank the technical reviewers who took time from their lives to give us feedback: Brooks Davis, John W. De Boskey, Alexey Dokuchaev, Julien Elischer, Pedro Giffuni, Marie Helene Kvello-Aune, Kurt Jaeger, Alexander Leidinger, Johannes Meixner, and Alexander Motin. We might not enjoy being told exactly how were wrong, but we do appreciate it.

Lucas would like to specifically thank iXsystems for their excellent test hardware, his wife Liz for everything, and Costco for their ultra-economy-size ibuprofen.

Jude would like to thank the *BSD community for welcoming him so warmly, with special thanks to his mentors Benedict Reuschling, Warren Block, and Eitan Adler, as well as Dru Lavigne, Devin Teske, George Neville-Neil, and Matt Ahrens.

Lucas would also like to thank specifically Jude for that effusive acknowledgement of specific cool people in the FreeBSD community, thus making him look churlish in comparison. But hes thanked specific FreeBSDers before, so it could be worse.

We dedicate FreeBSD Mastery: ZFS to our good friend

Paul Schenkeveld

who sadly passed away as we wrote this book

Much of our systems administration training focuses on filesystems. A computers filesystem dictates so much of its performance and behavior. Over the last decades weve rebuilt entire systems because a major filesystem was configured incorrectly, or the filesystem chosen wasnt suitable for the task, or because subtle filesystem corruption spread throughout our files and now we couldnt trust even the basic programs the operating system had shipped with. Anyone whos been a sysadmin more than a few years has learned how to repair filesystems, rebuild filesystems, work around bugs from vexing to nearly lethal, rearrange disks to support filesystem limitations, and swear extensively at filesystems in no fewer than nine languages.

Some of todays most popular filesystems are, in computing scale, ancient. We discard hardware because its five years old and too slow to be bornethen put a 30-year-old filesystem on its replacement. Even more modern filesystems like extfs, UFS2, and NTFS use older ideas at their core.

The Z File System, or ZFS, is here to change all that.

ZFS is a modern filesystem built with the core idea that the filesystem should be able to guarantee data integrity. ZFS computes a checksum for every piece of data on disk, so it can identify when the storage media has experienced an error and damaged the data. It performs the same cryptographic signatures on all of the metadata. Whennot ifthe underlying hardware has a problem or just misfires, ZFS realizes that the data it has retrieved doesnt match its records and can take action. ZFS even automatically corrects discovered errors! ZFS refuses to serve data it knows to be corrupt.

Filesystem designers had these ideas 30 years ago, but the hardware of the time couldnt perform this amount of error checking with reasonable performance. The creators of ZFS looked at current hardware as well as where hardware was going, and decided that ZFS would take full advantage of emerging hardware. The result is a filesystem thats not only more reliable than traditional filesystems, but often faster.

Today, it seems that traditional filesystems were written with a good enough for now philosophy. Many filesystems suffered from arbitrary size limits, which sufficed for five years, or ten, or even 20 but eventually required rewriting and reworking. Many older filesystems couldnt handle partitions larger than two gigabytes, which these days is smaller than a flash drive youll get for free attached to a bottle opener. (And really, you picked that up because you wanted the bottle opener.) But in the early 1980s, when UFS was first released, two gigabytes was a ridiculously large amount of storage that would cost many millions of dollars. Filesystems like FAT needed to efficiently use the space on 360 KB floppy disks. UFS was good enough for now, and for some time to come.

ZFS is deliberately designed to survive the foreseeable future and more. Many new filesystems use 64-bit identifiers and indexes internally, so theyll be usable without change for the next ten or 20 years. ZFS use 128-bit identifiers internally, giving it enough capacity to work on storage systems for the next several millennia. The Enterprises computer on Star Trek probably runs ZFS. Future sysadmins who must deal with disks, partitions, and files that exceed ZFS built-in constraints will lump us together in history with the cavemen and the first interstellar travelers.

Strictly speaking, ZFS is not just a filesystem. Its a combination filesystem and volume manager. Combining these two functions in one set of software does impose certain limitations, which well talk about laterbut it also makes some very interesting things possible. ZFS, being aware of exactly where data is going on the disk, can arrange files and stripes optimally, from top to bottom. ZFS can use secondary fast storage as special-purpose caches, further enhancing performance.

FreeBSD Mastery: ZFS Essentials takes you through what you must know to run this modern, high-performance, future-proof filesystem.

Matt Ahrens and Jeff Bonwick created ZFS for Sun Microsystems Solaris operating system. While Sun sold systems of all sizes, its main focus was high-end server hardware. Sun hardware drove most of the worlds large databases. Sun offered the ZFS source code to the world under its Common Development and Distribution License (CDDL). People began porting ZFS to other operating systems, including FreeBSD, and contributing changes back to Sun.

Then Oracle bought Sun Microsystems. While Oracle has some open source software, such as MySQL, most of its software is proprietary. Oracle brought ZFS development fully in-house and ceased providing source code under any open source license.