OpenZFS

Work in progress article

Zettabyte File System

What is ZFS?

If you don’t know, it’s a great time to learn. ZFS [1] was developed at Sun Microsystems in the early 2000s. ZFS is not just an on-disk data storage format like FAT or NTFS, but a complete storage software stack from stern to stem. Its purpose is applying the CIA triad to data. It is an alternative to RAID (both hardware and software).

The ZFS stack

Don’t mix it up with RAID or RAID controllers

Take snapshots

Replicate

Encrypt

Sources

Integrity of data at rest

ZFS hashes data nodes and metadata to identify unexpected changes and errors. By regularly conducting a scrub operation (re-hashing nodes and checking against stored values) ZFS can proactively and efficiently discover errors in data-at-rest.


Even if ZFS is used only on a single disk, where no physical redundancy exists, it still protects against silent corruption and bit rot. Few other filesystems implement this feature.


In combination with regular S.M.A.R.T. testing, ZFS can:

  • Detect errors in data-at-rest (most other filesystems cannot do this)

  • Track the frequency and physical location of errors

  • Alert administrators to impending drive failures

  • Automatically select a hot spare and rebuild a failed vdev

  • Supersedes the functions of mdraid/LVM

  • Provides better reliability than BTRFS

Comparison to RAID

ZFS replaces RAID by offering a variety of array structures to protect against device failure. These include modes analagous to RAID 0 (stripe), RAID 1 (mirror), RAID 5 (single parity), and RAID 6 (distributed parity). When a zpool is constructed atop a group of mirror vdevs, it is analagous in function to RAID 10/1+0.

What is a vdev?

A vdev is a virtual device. ZFS abstracts all physical devices as vdevs. A vdev can be any of the following:

  • Single drive
  • Stripe
  • Mirror
  • RAIDZ

RAIDZ has 3 levels: RAIDZ1, Z2 and Z3. The levels describe parity striping:

  • RAIDZ1 has 1 parity stripe and can tolerate any single drive failure.
  • RAIDZ2 has 2 parity stripes and can tolerate any two simultaneous drive failures.
  • RAIDZ3 has 3 parity stripes and can tolerate any three simultaneous drive failures.

What is a zpool?

A zpool is the root-level namespace for data. It is an array of virtual devices (vdevs), grouped logically. zpools can be constructed from multiple vdevs. This allows for the creation of highly flexible zpools that perform similarly in principle to nested or hybrid RAID, but with major performance benefits.


For example, a common zpool architecture is a composition of two 6-wide RAIDZ2 vdevs.


Data is striped across both vdevs, which yields performance benefits. Data is also distributed among disks within each vdev (except for mirror vdevs, which yield only read gains).


So more non-mirror vdevs = more IOPS.


But more vdevs =/= more protection. The survivability of the entire pool is determined by the survivability of the weakest vdev. Each individual RAIDZ2 vdev can survive two simultaneous disk failures. In theory, the entire pool can survive 4 simultaneous disk failures (2 in each vdev), but it could not survive the loss of 1 vdev.