[nylug-talk] dig weird behavior

[Frank D. Greco]

At 12:08 AM 9/28/2006, Michael Bacarella wrote:
>On Wed, Sep 27, 2006 at 11:44:25PM -0400, Frank D. Greco wrote:

> > Worked fine as expected.  But since my $HOSTS has been getting quite
> > large, I attempted to parallelize the guts of the loop:
> >
> >       for i in $HOSTS
> >       do
> >               dig ... | grep ... > $TMPDIR/$i   &
> >       done
> >
> >       wait
> >       cat $TMPDIR/* | sort
> >
> > Surprisingly, this hung.  When I checked my $TMPDIR, all the files
> > were created, but some were 0 bytes.  When I hit CTRL-C on this script,
> > I got some odd error about sockcount == 0 from dig.c
> >
> > Anyone know what's going on here?
>
>The shell creates the $TMPDIR/$i files before it even executes the
>processes on the command line; it's possible you're just not waiting
>long enough for them all to finish.

         That's what I thought too.  So I monitored the files in that $TMPDIR.
         Only some of the files were completed; the others were 0 bytes.
         I let it simmer for at least 30 min while I read about the new
         Intel quad-cores in EET... ;)

>If they're not finishing, you might be hitting some resource limit
>(ulimit -a) that causes dig to hang.  Or maybe there's a concurrency
>issue with having that many digs running.

         That's my guess too.  It feels like a concurrency issue with
         the digs.  You might be right about hitting a resource limit.
         But I am on a 8GB RAM box with 8GB of swap.  Dig can't be that
         big, can it?  I'll check this out.

         But I would expect a 'cannot fork' if I ran out of memory though...

>How big's the host list?

         About 50.  Not humongous.

>You could be running out of ephemeral ports.
>
>Don't think this stuff is robust just because it runs important
>internet infrastructure. ;)

         :)

         I've been working with RH Linux the past 3 years.  I would not
         use the word 'robust' in the same sentence.  ;)

         Thanks for the reply!

         Frank G.
--
+==============================================+
| Crossroads Technologies Inc.                 |
| www.CrossroadsTech dot com                   |
| fgreco at REMOVEtheX!cross!roads!tech!dotXcom|
+==============================================+
--