|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
[PLUG] perplexing file transfer problem
|
I've encountered a file transfer problem that is quite perplexing -
and I'm at a loss to identify the cause. Part of the problem is
that it's random and it's complex (a number of variables are
involved) and therefore hard to narrow down to a single failure
point.
The basic scenario is this: I have a shell script that uses curl
commands to retrieve multiple files from two ftp servers. The
host for this is a rented box that I do not have root or physical
access to. I do not have root or physical access to the ftp
server either for that matter... I'm just a user/developer
tryin' to bring the data home :-)
The script ran on a shared box for several months but was moved to
it's own box with a new IP on Thursday. On Friday the random
failures began.. some initial portion of the file would download at
normal speed then the rate would drop to zero bytes per second and
stay there. If the script is re-run it may download that file just
fine only to fail on the next download. Also, the size of the
initial download appears random too - one was 292 kB, the next was 1.9
Meg... etc. There is a log fragment at the bottom of this email
showing the relevant part (the output of the curl command).
Other factors I've considered:
The transfer randomly fails if we use ftp from the command line too
- it's not just curl
For security reasons, the ftp server must have the IP of the
requestor on file - and it does. Otherwise, we could not
even log it.
The failure can be from either of the ftp servers - although they
are on the same domain they are separate host names.
Now if anybody has seen something similar or has any ideas on how to
further troubleshoot this I'd be very grateful. Keep in mind that it's
been working like clockwork since March or April so it's not the
programmer's fault :-D
Thanks.
Eric
Our host:
# uname -a
Linux host.xxxxxxxxx.com 2.6.9-34.EL #1 Wed Mar 8 00:07:35 CST 2006
i686 athlon i386 GNU/Linux
Log fragment showing how it fails:
-----
curl -n $DATAURL1 > $DATAFILE1
retrieving BR_20060710_013000__dat_full_.zip
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 24.1M 0 1460 0 0 1283 0 5:29:00 0:00:01 5:28:59 1283
0 24.1M 0 168k 0 0 106k 0 0:03:53 0:00:01 0:03:52 371k
2 24.1M 2 684k 0 0 264k 0 0:01:33 0:00:02 0:01:31 470k
4 24.1M 4 1166k 0 0 319k 0 0:01:17 0:00:03 0:01:14 462k
4 24.1M 4 1199k 0 0 219k 0 0:01:52 0:00:05 0:01:47 277k
4 24.1M 4 1199k 0 0 185k 0 0:02:13 0:00:06 0:02:07 225k
4 24.1M 4 1199k 0 0 160k 0 0:02:33 0:00:07 0:02:26 175k
4 24.1M 4 1199k 0 0 141k 0 0:02:54 0:00:08 0:02:46 89742
4 24.1M 4 1199k 0 0 126k 0 0:03:15 0:00:09 0:03:06 5781
4 24.1M 4 1199k 0 0 114k 0 0:03:35 0:00:10 0:03:25 0
4 24.1M 4 1199k 0 0 104k 0 0:03:56 0:00:11 0:03:45 0
4 24.1M 4 1199k 0 0 98511 0 0:04:17 0:00:12 0:04:05 0
4 24.1M 4 1199k 0 0 91188 0 0:04:37 0:00:13 0:04:24 0
4 24.1M 4 1199k 0 0 84879 0 0:04:58 0:00:14 0:04:44 0
.
. [five hours later we see.... Current Speed is stuck at zero]
.
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:55 99:56:12 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:56 99:56:11 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:57 99:56:10 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:58 99:56:09 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:59 99:56:08 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:00 99:56:07 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:01 99:56:06 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:02 99:56:05 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:03 99:56:04 0
4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:04 99:56:03 0
--
------------------------------------------------------------------------
# Eric A Lucas
# ------------
# "Oh, I have slipped the surly bond of earth
# and danced the skies on laughter-silvered wings...
# -- John Gillespie Magee Jr.
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|