[nylug-talk] Big filesystem recommendation - which one?

[Brandorr]

On 8/21/07, Steven Lembark <lembark at wrkhors.com> wrote:
> David Rosenstrauch wrote:
> > jh wrote:
> >> I'm getting ready to build a new fileserver. Hardware raid, about a
> >> dozen HDs, about 5TB of storage. NFS and Samba file serving duties.
> >>
> >> My peculiar needs are that my work entails huge populations of
> >> tiny-to-small files - for example, 20 million TIFF images in a
> >> collection, each between 5 and 500k in size.
> >>
> >> The last file server I built used XFS on a 1.5TB filesystem (this was a
> >> few years ago, when 1.5TB was still slightly impressive), and it worked
> >> pretty well - hell, it still works great. Deletions are a bit pokey with
> >> XFS it seems, but that's my only complaint.
> >>
> >> What would you use today?
> >>
> >> jh
> >
> > I've read that ReiserFS works well with large numbers of files.
>
> RFS is nice for huge nubmers of small files (think
> imap4 server); XFS handles larger files more
> gracefully with less CPU overhead. There are a few
> good benchmark articles floating around.
>
> I've used XFS without any problems on high-volume
> filesystems and ones with really large files. The
> realtime volume can be a big performance help also.

Both XFS and ReiserFS have data integrity issues. XFS is tuned for
fast sequential IO of large files.

http://www.novell.com/documentation/suse91/suselinux-adminguide/html/apas02.html

I've been doing some more digging on ZFS. It is probably the best file
system for your use case. (Space utilization is very efficient for
small files, data integrity is probably the best out there, and it's
all set to scale to whatever file system size you would need in the
future.).

The only issue is whether or not your hardware has drivers. (HG and I
linked you to the Java tool to test your hardware.)

If your hardware is supported, I'd be interested in helping you set
this up. (After hours and weekends only). The only thing I ask is that
you let me run some benchmarks on your setup, and test some
configuration tunables..... (If you don't have time to let me do this,
I still want to help)

Cheers,
Brian

P.S. - I checked out Linux compatibility. If you are running recent
kernels, you shouldn't have any issues performance or otherwise.

-- 
- Brian Gupta

http://opensolaris.org/os/project/nycosug/