[nylug-talk] The best way to mirror 6M+ files?

[alex@pilosoft.com]

On Mon, 8 May 2006, Yusuke Shinyama wrote:

> Actually one of the top-level directories has already 3M+ files and the
> structure of the subdirectories might change. Maybe I need to ask the
> users to organize them in a more consistent way.
In this case, you have already failed. It is a very bad idea to have >10k 
files in same directory, some things will start acting very strange. For 
example, if you ever have to fsck that filesystem, expect it to never 
complete at best, or corrupt that directory at worst. 

> > If you need to have the files be stable while you do this, I suggest
> > you look at the volume managers snapshot capabilities.
> 
> They don't have to be stable, but the source disk needs to be accessible
> all the time. The users run various experiments on those files via NFS.
Consider dump/restore, since they do backup inode-by-inode without looking 
at the directory contents, and thus not affected by number of files in it.