ZeroFS: log-structured filesystem for S3

2 min read
self-hostingopen-sourceinfrastructures3
View as Markdown
Originally from zerofs.net
View source

My notes

Summary

ZeroFS is a log-structured filesystem that exposes S3-compatible object storage (AWS S3, GCS, Azure Blob, or any S3-compatible store) as a POSIX filesystem over NFS and 9P, and as raw block devices over NBD, all from one userspace process. It turns cheap object storage into something ZFS pools, VM boot disks, and normal file operations can run directly on top of, with encryption and compression built in.

Key Insight

  • Data is split into 32 KiB extents packed into immutable segment objects up to 256 MiB, a log-structured design that avoids in-place rewrites in object storage.
  • Every extent gets XChaCha20-Poly1305 encryption before upload, with the data key wrapped by an Argon2id-derived key from a password, encryption is not optional or bolted on.
  • Compression (zstd or lz4) is applied before encryption, and the codec can be changed at any time without a migration step, existing data’s codec is auto-detected on read.
  • The three protocols (NFS, 9P, NBD) have different durability guarantees: NFS COMMIT lets fsync return before data actually reaches S3, while 9P and NBD only acknowledge fsync/FLUSH/FUA after the write is durable in the object store. This matters for anyone assuming fsync means “safe”, on NFS it doesn’t.
  • The bundled FUSE 9P client (zerofs mount) needs no root and auto-reconnects, positioned as the recommended Linux mount over raw NFS.
  • Supports a genuine ZFS mirror across three separate S3 regions, each ZeroFS instance exposes one region’s bucket as a block device, and ZFS treats them as plain disks, giving object-storage-backed geographic redundancy without a SAN.
  • Read replicas: one writer instance plus any number of read-only replicas against the same bucket; replicas follow the writer and return EROFS on write attempts.
  • Failover: an optional standby holds acked-but-unflushed writes and can take over writer duties within seconds.
  • Test rigor is unusually heavy for an infra project: 8,662 pjdfstest POSIX compliance cases, xfstests, full Linux kernel parallel compiles running on top of NFS/9P/FUSE mounts, stress-ng file-handling stressors, a ZFS pool+scrub test built on ZeroFS block devices, and Jepsen tests (including a leader/standby HA suite that kills and pauses nodes against a local S3 server).