Choosing The BEST Drive Layout For Your NAS

Summary: NAS drive layout for ZFS (4–8 drives)

The video explains how to choose NAS drive layouts in ZFS-based systems (explicitly TrueNAS Core, with general relevance also for TrueNAS SCALE and even Unraid for broader concepts). It focuses on:

• How many drives to use
• How to configure them using pools and vdevs
• Comparisons between mirrored, striped, and RAID-Z layouts (RAID-Z1/Z2/Z3)

---

What the tutorial covers (key topics)

1) ZFS fundamentals: pools and vdevs

• Pools are the top-level storage containers where datasets/volumes live.
• Pools are composed of vdevs (virtual devices). Each vdev is treated by the pool as a “virtual drive.”
• ZFS spreads data across vdevs, and vdevs can be built as:
  • Stripe
  • Mirror
  • RAID-Z (RAID-Z1/Z2/Z3)

2) RAID/ZFS configuration concepts explained

Striped vdev

• No redundancy (parity effectively none)
• Uses all drive capacity
• Data loss if any single drive fails

Mirrored vdev

• Data is duplicated identically across drives
• Two-way mirrors: usable capacity ~ half
• Three-way mirrors (not deeply benchmarked): usable capacity ~ one-third
• Resilience caveat: losing two drives in the same vdev is unrecoverable

RAID-Z

• Similar to RAID-5/6, but with important ZFS-specific differences
• RAID-Z1: 1 drive worth of parity per vdev; tolerates 1 drive failure per vdev
• RAID-Z2: 2 drive worth of parity per vdev; tolerates 2 drive failures per vdev
• RAID-Z3 mentioned but not covered deeply (3 parity)

3) Trade-offs (“triangle”) + cost factor

The core trade-offs are framed as optimizing:

• Capacity
• Resilience
• Performance

Typically, you can optimize two at the expense of the third.

A fourth factor is cost, described as an “upside-down pyramid,” where spending more can improve capacity/resilience/performance.

4) Guidance on scaling from 4 → 6 → 8 drives

The video compares performance/resilience as drive counts widen, emphasizing:

• Wide vdevs (e.g., very wide RAID-Z vdevs) can raise risk because the chance of multiple failures during rebuild increases.
• Often, it’s better to use multiple smaller vdevs (e.g., multiple RAID-Z1 vdevs or multiple mirror vdevs) rather than one extremely wide vdev.

---

Theoretical examples (capacity + resilience + performance)

Four-drive layouts (examples)

Using 1 TB drives for easy math:

• Stripe: ~4 TB usable, but 0 redundancy
• RAID-Z1: ~3 TB usable (1 TB parity)
• RAID-Z2 and two mirrors: ~2 TB usable (2 TB parity / mirror halves usable space)

Resilience notes

• If a vdev fails, the entire pool is lost (the unit of failure is the vdev, not the whole disk set).
• Resilvering time matters: during rebuild, another failure can be catastrophic.
• ZFS resilvering rebuilds only the data that existed on the failed drive, unlike some assumptions in traditional RAID rebuild models.
• Mirrors tend to resilver faster than wider RAID-Z/Z pools (referenced, not deeply benchmarked).

Performance notes (simplified expectations)

• Striping: bandwidth/iops scale with all drives
• Mirrors: reads scale like multiple drives; writes limited by vdev count
• RAID-Z: each vdev behaves like roughly a single-drive IOPS unit, while streaming bandwidth is reduced by parity overhead (e.g., RAID-Z1 loses ~1 drive worth of bandwidth)

---

Real-world benchmarking (review/analysis)

Test platform

• Borrowed TrueNAS Mini R server
• 12 hot-swap bays
• 10GbE networking
• Workloads executed on TrueNAS Core
• Drives: 10 TB Western Digital Red Plus
• Hardware:
  • Intel i9-9900H
  • NVMe SSD client side
  • Client uses Debian 12

Workloads / tools

• fio (random + sequential tests)
  • Random: 64K blocks, depth 8, 8 threads
  • Sequential: 256K blocks, depth 1, single thread
• disbench (file transfer time)
  • Copy a 20 GB video file multiple times (large sequential-ish transfer)
  • Then copy a folder of many small config/text files (intended to stress random-ish I/O)

Arc cache handling

• Tests run twice:
  1. With ZFS ARC disabled
  2. With ARC enabled
• Key takeaway:
  • ARC didn’t drastically change results because workloads weren’t large enough to overwhelm ARC.
• Most conclusions rely on ARC disabled results.

---

Benchmark findings (what performed best)

Sequential performance (fio, ARC disabled)

• Striped pools: best raw sequential bandwidth, but no redundancy
• Mirrors: strong sequential read/write compared to RAID-Z in some cases
• With more drives:
  • At 6 and 8 drives, two RAID-Z vdevs appeared to offer a good balance of read and write performance based on observed results
  • Some results were counter to expectations (e.g., mirrors not always leading in reads for certain setups)

Scaling behavior

• “Overall scale well with number of drives,” but the bump from 6 → 8 drives wasn’t as large as anticipated.
• Often, the biggest performance differentiator across tests was the number of vdevs, not just the drive count.

Random I/O behavior

• Random tests emphasized IOPS vs bandwidth
• Conclusion: more vdevs → more IOPS (consistent with theory and partially supported by results)
• Random write results were described as “messy,” possibly due to:
  • ZFS memory/transaction group effects
  • Test not reflecting drive capability cleanly

disbench (Windows file transfer)

Large file transfers (20 GB video)

• With 4-drive comparisons: striped mirrors faster than RAID-Z
• With 6/8 drive comparisons: wider RAID-Z vdevs reportedly performed best

Small file transfers (many small text/config files)

• Did not behave as expected.
• Suspected causes: caching and transaction groups smoothing results, plus potential limitations from SSD/network.
• Considered less reliable than other tests.

---

Practical recommendations (use cases)

• Video/streaming-focused NAS (less concerned with huge operations/sec)

  • Suggest: 2× vdevs of 4 drives in RAID-Z1 or RAID-Z2

• Mission-critical backups

  • Suggest: RAID-Z2 or RAID-Z3

• Multi-user / heavy small-I/O (needs lots of IOPS)

  • Suggest: mirrors (striped mirrors mentioned) as potentially “no-brainer” for IOPS-heavy workloads
  • For extra resilience: three-way mirrors

• Regardless of layout:

  • Keep backups on a separate system
  • Hardware failure and human error are emphasized as real risks

---

Main speakers/sources

• Main speaker/source: The video author/host (Hardware Haven; no specific person name given in the subtitles)
• Sponsored source mentioned: HelloFresh
• Benchmark/reference sources mentioned:
  • ex systems (provided the TrueNAS Mini R and helped with benchmarks)
  • lauren.com (benchmarks linked in description for resilver performance comparisons)
  • ex systems PDF (theoretical performance limitations of pool configurations)
  • Additional documentation/resources referenced in the description (e.g., caching datasets)