[nylug-talk] Big filesystem recommendation - which one?

[jh]

Brandorr wrote:
> On 8/21/07, Steven Lembark <lembark at wrkhors.com> wrote:
>> David Rosenstrauch wrote:
>>> jh wrote:
>>>> I'm getting ready to build a new fileserver. Hardware raid, about a
>>>> dozen HDs, about 5TB of storage. NFS and Samba file serving duties.
>>>>
>>>> My peculiar needs are that my work entails huge populations of
>>>> tiny-to-small files - for example, 20 million TIFF images in a
>>>> collection, each between 5 and 500k in size.
>>>>

> I've been doing some more digging on ZFS. It is probably the best file
> system for your use case. (Space utilization is very efficient for
> small files, data integrity is probably the best out there, and it's
> all set to scale to whatever file system size you would need in the
> future.).

I'd agree that ZFS could be a better fit than existing Linux filesystems.

> 
> The only issue is whether or not your hardware has drivers. (HG and I
> linked you to the Java tool to test your hardware.)

The hardware hasn't been purchased yet, so I'm open to suggestions, 
although I do have my shopping list...

http://www.tyan.com/product_board_detail.aspx?pid=349
Tyan Intel Mobo - if this isn't compatible, well, I'll won't be looking 
back while I'm loading Linux :)

RAID/JBOD controller will be:

http://www.3ware.com/products/serial_ata2-9650.asp

I'm guessing that this is, in fact, a problem. I also know that you 
don't need fancy RAID hardware for ZFS, right? That'd I'd be better off 
with a bunch of non-raid SATA ports...not sure that I can get a few 
extra ports into the budget.
> 
> If your hardware is supported, I'd be interested in helping you set
> this up. (After hours and weekends only). The only thing I ask is that
> you let me run some benchmarks on your setup, and test some
> configuration tunables..... (If you don't have time to let me do this,
> I still want to help)

I'm game, once I get funding for the new server (should be imminent). 
Perhaps this should go offline...

jh