[nylug-talk] questions about /dev/null

[Peter C. Norton]

On Mon, Nov 05, 2007 at 07:30:29AM -0400, Sunny Dubey wrote:
> On Monday 05 November 2007 02:04:03 am Judd Maltin wrote:
> > Is there any reason writing a stream of ASCII to /dev/null would be
> > faster than copying the same file to /dev/null ?
> >
> > And then int he context of MySQL, why would it be faster to do a
> > mysqldump into /dev/null than to do a mysqlhotcopy into /dev/null -
> > faster by 1/2 the time.
> >
> > Odd, odd. ?Is it inodes? ?For some reason not reading smoothly across
> > the disk?
> 
> This is a common misconception.
> 
> /dev/null is a *file*.  It is a special file and not a regular one:
> 
> sunny at localhost ~ $ file /dev/null
> /dev/null: character special (1/3)

Yes, and as such it still has inodes, etc.
 
> Writing *TO* /dev/null causes no disk activity because you are writing to a 
> file that simply discards the information.  Writing *ON* /dev/null causes 
> disk activity because now you have replaced the file with something else that 
> does not discard the information, and actually grows with whatever you are 
> writing on it.

I'm not sure you're on the right track here.  A regular user can't
unlink and overwrite /dev/null, which is always owned by root in a
directory that is owned by root and is not world-writeable. This means
that the directory entry can't be removed and subsequently replaced by
a normal user. Even in the root user case, the open() system call will
open an existing file. It will not unlink() and then create a new
file. I think the answer is a bit different...

> You can think of /dev/null as a garbage pail with a bottomless pit.  You could 
> throw everything at it, and it would simply "disappear".  But the moment you 
> replace it with not something special, a garbage mound would start to grow.

This usually can't happen (if it ever does happen on a system, it's
probably by some combination of deliberate and stupid and malicious),
and many, many things would break once the filesystem under /dev
filled up.

The usual reason for writes to /dev/null happening faster with some
programs and scripts than with others is that the C library or the
system may treat /dev/null very specially. If it knows that it's
supposed to write to the null device, it will probably short-circuit
the process and simply free the memory that would have been written.

*However* when a program has to write to a pipe or to stdout/stderr
which *then* gets dumped to /dev/null, then the program has to first
do the write, and the reading end of the pipe has to read, and errors
checked, etc.

Something I did recently showed this off:

First, prime the read-cache:

spacey at spacey:~$ sudo dd if=/dev/md0 of=/dev/null bs=1024k count=100
Password:
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.81393 seconds, 57.8 MB/s

Second, re-run with the same space in cache:

spacey at spacey:~$ sudo dd if=/dev/md0 of=/dev/null bs=1024k
count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.579875 seconds, 181 MB/s

Run again, this time with ">", which still gives the process a file
descriptor that leads to /dev/null:

spacey at spacey:~$ sudo dd if=/dev/md0  bs=1024k count=100 >
/dev/null
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.577291 seconds, 182 MB/s

Now watch what happens when we put a process in the middle, even a
stupid process, that has to read the pipe before throwing out the
data:

spacey at spacey:~$ sudo dd if=/dev/md0  bs=1024k count=100 | cat >
/dev/null
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.79849 seconds, 58.3 MB/s

To show it's not a fluke:

spacey at spacey:~$ sudo dd if=/dev/md0  bs=1024k count=100 | cat >
/dev/null
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.72081 seconds, 60.9 MB/s

So that's the slowdown for a simple, simple, simple program like "cat"
to read buffere stdin and to throw it out. A >50% slowdown. The kernel
or the C library simply has to stat() the file being written to, and
see if it matches the st_rdev of /dev/null on the current system (yes,
it's not codified anywhere that these device numbers must be uniform
across platforms or releases).

Disclaimer: no /dev/nulls were harmed during the filming of this
demonstration.

-Peter

P.S. This is a common optimization on all unix platforms that I'm
aware of.

-- 
The 5 year plan:
In five years we'll make up another plan.
Or just re-use this one.