[nylug-talk] Filesystems that Support Transparent Compression?

[Michael Bacarella]

On Tue, Jul 11, 2006 at 06:35:19PM -0400, alex at pilosoft.com wrote:
...
> > > Today I was asked if we can use any filesystem on Linux that
> > > supports transparent compression. Because we're trying to handle
> > > hunderds of gigabytes of XML data that can be compressed very well
> > > (about 80-90% size reduction), we could save a lot of disk space
> > > by using such a filesystem.
> > 
> > I've never understood why a feature which became standard with PC
> > operating systems in the 1980's has yet to become standard with Linux. I
> > tend to do more long-term storage on my Windows boxes than my Linux
> > boxes precisely because the Windows boxes allow me to more cheaply store
> > the data.
> Because this isn't compatible with unix sematics for filesystem, and 
> doesn't make sense.
> 
> Example: You have 20GB file, compressed into 10G. Now, you mmap'd it, and 
> need to fetch byte #44444444. To do that, you need to uncompress entire 
> file. Now, you modify a byte there. To write that, you need to recompress 
> entire file. Etc.

If the file were broken into 8192 byte chunks with each chunk compressed and
packed together, add an index which shows the mappings, I could see this
working pretty well.  You maintain random accessibility features and possibly
improve sequential read performance since one physical block read may yield
several blocks worth of data.