3d ago

What's a reliable filesystem for long term data archival that also supports transparent file compression?

I currently use btrfs on almost all my server and offline backup drives mainly because of the ability to transparently compress files to save space. But apparently btrfs can get corrupted if it unexpectedly loses power? Is there a more robust and corruption resistant filesystem that you'd recommend that also has the ability to compress files?

15 comments

Corruption on power loss is something that journaling should prevent and modern filesystems do that. So btrfs should be as safe as other modern filesystems.
Edit: Btrfs does no journaling, it implements resilience with another mechanism, namely Copy on write.
- It should, but apparently it isn't, at least not reliably.
  
  It's perfectly reliable. You only need caution when using RAID5 and RAID6 as these features are still in the experimental phase.
  
  I've lost power with it many times, none of my BTRFS partitions have been corrupted so far.
I believe ZFS fits your needs. Using ZFS with TrueNAS I can verify it supports transparent compression. It's supposed to be very stable against data corruption. The only thing I can say about data corruption is in my 10+ years of using it, I haven't had any issues.
- For the OP, in case you go for zfs, don't forget to create a cron job for scrubbing, truenas has support for this built in. It repairs things like bit rot, should they occur.
  
  Good looking out.
- It never had data corruption issues in 30+ years on EXT4 either... It doesn't mean that you had none.
  
  ZFS has built in error checking and correction if you have parity data. Its less if they have had corruption and more that it doesnt matter, because their system is designed to correct it automatically anyway.
  With EXT4 over 30 years, you've likely been lucky or unaware of any corruption. Guess thats the same thing in the end.
But apparently btrfs can get corrupted if it unexpectedly loses power?
Afaik this is only a problem if you use RAID 5 or 6. Otherwise it's as solid as other CoW filesystems like zfs.
https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices
ZFS
Are you using RAID at all? If so ZFS is probably the way to go. If not I think it matters less whether you use either btrfs or ZFS.
Regarding btrfs and power loss:
https://unix.stackexchange.com/questions/340947/does-btrfs-guarantee-data-consistency-on-power-outages
...btrfs is designed to only experience data loss not corruption, assuming well behaving hardware in power outage scenario. In practice ZFS has more maturity overall (definately) so may be better (my speculation).
Beyond direct comparisons if you already have on and offline backups then you are protected from power corruption and only have to worry about data loss anyway?
If you're using consumer HDD's or SSD's then they themselves can get corrupted in power failures, so the software almost doesn't matter. Better use a UPS.
I think for home archival use is btrfs is a terrible idea. I burnt myself with it so many times.
Yes it should be safe and resistant to unclean shutdowns, but my experience with reliability is not great.
apparently btrfs can get corrupted if it unexpectedly loses power?
What do you mean by that?

15 comments