[nylug-talk] The best way to mirror 6M+ files?

[Steven Lembark]

 > Actually one of the top-level directories has already 3M+ files
 > and the structure of the subdirectories might change. Maybe I need to
 > ask the users to organize them in a more consistent way.
 >
 >> If you need to have the files be stable while you do this, I suggest
 >> you look at the volume managers snapshot capabilities.
 >
 > They don't have to be stable, but the source disk needs to be
 > accessible all the time. The users run various experiments on
 > those files via NFS.

At some point it becomes your option as SysAdmin to
impose an order in order to avoid chaos or loss of
data. Seems like you've hit both by now...

Mirroring the drive will be the best solution for a
stable system since it is a background operation.
Main problem will be that if people are hitting the
filesystem faster than Linux can mirror it your
mirror will never finish. Shutting down for an hour
at 0300 on a Sunday is a reasonable fix for this.

There must be SOME organization to the files: type,
owner... I'd suggest finding it and imposing some
sort of order quickly before you do have to fsck the
system and find the heat death of the universe quickly
approaching as you hit 50% complete...


-- 
Steven Lembark                                         85-09 90th Street
Workhorse Computing                                  Woodhaven, NY 11421
lembark at wrkhors.com                                      +1 888 359 3508