[nylug-talk] The best way to mirror 6M+ files?

[Yusuke Shinyama]

alex at pilosoft.com wrote:
> > Actually one of the top-level directories has already 3M+ files and the
> > structure of the subdirectories might change. Maybe I need to ask the
> > users to organize them in a more consistent way.
> In this case, you have already failed. It is a very bad idea to have >10k 
> files in same directory, some things will start acting very strange. For 
> example, if you ever have to fsck that filesystem, expect it to never 
> complete at best, or corrupt that directory at worst. 

Tell me about it.  I didn't expect this at the beginning, but then
they started blindly putting everything into it...  They never
understand there is a limitation. Sigh.

> > > If you need to have the files be stable while you do this, I suggest
> > > you look at the volume managers snapshot capabilities.
> > 
> > They don't have to be stable, but the source disk needs to be accessible
> > all the time. The users run various experiments on those files via NFS.
> Consider dump/restore, since they do backup inode-by-inode without looking 
> at the directory contents, and thus not affected by number of files in it.

I'll try it. Maybe I should've considered switching to raid-1 earlier,
before things get so bad.

Thanks,
Yusuke