Re: [PLUG] curl error

[Eric <eric@lucii.org>]

Brian Vagnoni wrote:
> Have you tried a -0 -S or -s -N options.
> 
> Brian Vagnoni

Interesting idea - particularly the -N to disable buffering and the -S to show
errors.  I'll add those to see if it helps.

Remember that this process runs forever so the error is, as far as I know,
outside of curl.

Here's what happens in a nutshell:

In the bash script the command is executed:

   curl -n $DATAURL1 > $DATAFILE1

The script output is logged so the log file records the little progress display,
reproduced here:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  642M    0  1460    0     0   1997      0 93:42:23 --:--:-- 93:42:23  1997

This tells me that the file is 642 meg in size and that curl is downloading at
the rate of 1997 bytes/second.  It estimates 93 hours, 42 minutes to complete.
For reference, I've routinely seen 800k bytes/second between the servers so
we're off to a slow start :-(


After a six seconds it has picked up:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  642M    0 3528k    0     0   582k      0  0:18:49  0:00:06  0:18:43  673k

Note the average download speed is 582k - nice clip
Seven + minutes later we're 45% of the way done:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 45  642M   45  293M    0     0   675k      0  0:16:13  0:07:24  0:08:49  676k

Oops, less than a minute later the transfer mysteriously stops (zero in last
column):

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 46  642M   46  300M    0     0   627k      0  0:17:28  0:08:09  0:09:19     0

Much later, I end up killing the shell script and the curl process:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 46  642M   46  300M    0     0   3648      0 51:17:49 23:57:37 27:20:12

Note:
   It's almost 24 HOURS later - curl did not exit or error out... it
      simply keeps on running... for days if not interrupted.

   No data was received since 8:09 into the process.

I'm thinking that maybe the server ftp process dies and leaves the port open
thereby fooling the network into leaving the connection open?  I'm not a network
guru so that's a SWAG.


Thanks,
Eric
-- 
#  Eric Lucas
#
#                "Oh, I have slipped the surly bond of earth
#                 And danced the skies on laughter-silvered wings...
#                                        -- John Gillespie Magee Jr
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug