[nylug-talk] Filesystems that Support Transparent Compression?

[Ron Guerin]

Yusuke Shinyama wrote:

> Today I was asked if we can use any filesystem on Linux that
> supports transparent compression. Because we're trying to handle
> hunderds of gigabytes of XML data that can be compressed very well
> (about 80-90% size reduction), we could save a lot of disk space
> by using such a filesystem.

I've never understood why a feature which became standard with PC
operating systems in the 1980's has yet to become standard with Linux.
I tend to do more long-term storage on my Windows boxes than my Linux
boxes precisely because the Windows boxes allow me to more cheaply
store the data.

> I found there is a patch for ext2, but this looks kinda outdated
> and not very reliable. I've also read ReiserFS4 and ZFS supports
> compression. But I'm not sure if I can go with Reiser4 either...
> ZFS is not ported to Linux yet. Does anyone have any experience of
> this? Is there any other option? We do care the reliability and
> stability of software, but the speed is not a big issue.

I'd stay away from a patched filesystem.  Here today, gone tomorrow.
Could be the patch, could be your data, possibly both.  Just the pain of
having to stop using it is enough to stay away in the first place.

> Personally I think application-level compression is still better
> choice. Then I would be interested in there's any wrapper or
> library that facilitate this sort of integration.

I run a specialized DNS server called rbldnsd .  It has read plain text
files for input data, but recently in a new version, it has started
reading gzipped files, doing on the fly decompression.  I believe the
only dependency added was zlib.  Perhaps something like that would be
useful for you.  It is "semi-transparent" if you will, because a typical
*nix user is already well equipped to deal with gzipped files, should
you want to do something with them outside of your application.

- Ron