|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|
Re: [PLUG] perplexing file transfer problem
|
It turned out that the provider had their firewall ports closed...
opening that up fixed the problem... the transfer rate is about
1 mByte/second.... sweet!
Thanks for the suggestions.
Eric
On Monday 10 July 2006 10:02 am, Eric wrote:
> I've encountered a file transfer problem that is quite perplexing -
> and I'm at a loss to identify the cause. Part of the problem is
> that it's random and it's complex (a number of variables are
> involved) and therefore hard to narrow down to a single failure
> point.
>
> The basic scenario is this: I have a shell script that uses curl
> commands to retrieve multiple files from two ftp servers. The
> host for this is a rented box that I do not have root or physical
> access to. I do not have root or physical access to the ftp
> server either for that matter... I'm just a user/developer
> tryin' to bring the data home :-)
>
> The script ran on a shared box for several months but was moved to
> it's own box with a new IP on Thursday. On Friday the random
> failures began.. some initial portion of the file would download at
> normal speed then the rate would drop to zero bytes per second and
> stay there. If the script is re-run it may download that file just
> fine only to fail on the next download. Also, the size of the
> initial download appears random too - one was 292 kB, the next was 1.9
> Meg... etc. There is a log fragment at the bottom of this email
> showing the relevant part (the output of the curl command).
>
> Other factors I've considered:
>
> The transfer randomly fails if we use ftp from the command line too
> - it's not just curl
>
> For security reasons, the ftp server must have the IP of the
> requestor on file - and it does. Otherwise, we could not
> even log it.
>
> The failure can be from either of the ftp servers - although they
> are on the same domain they are separate host names.
>
> Now if anybody has seen something similar or has any ideas on how to
> further troubleshoot this I'd be very grateful. Keep in mind that it's
> been working like clockwork since March or April so it's not the
> programmer's fault :-D
>
> Thanks.
> Eric
>
> Our host:
>
> # uname -a
> Linux host.xxxxxxxxx.com 2.6.9-34.EL #1 Wed Mar 8 00:07:35 CST 2006
> i686 athlon i386 GNU/Linux
>
> Log fragment showing how it fails:
>
> -----
> curl -n $DATAURL1 > $DATAFILE1
> retrieving BR_20060710_013000__dat_full_.zip
> % Total % Received % Xferd Average Speed Time Time Time
Current
> Dload Upload Total Spent Left
Speed
> 0 24.1M 0 1460 0 0 1283 0 5:29:00 0:00:01 5:28:59
1283
> 0 24.1M 0 168k 0 0 106k 0 0:03:53 0:00:01 0:03:52
371k
> 2 24.1M 2 684k 0 0 264k 0 0:01:33 0:00:02 0:01:31
470k
> 4 24.1M 4 1166k 0 0 319k 0 0:01:17 0:00:03 0:01:14
462k
> 4 24.1M 4 1199k 0 0 219k 0 0:01:52 0:00:05 0:01:47
277k
> 4 24.1M 4 1199k 0 0 185k 0 0:02:13 0:00:06 0:02:07
225k
> 4 24.1M 4 1199k 0 0 160k 0 0:02:33 0:00:07 0:02:26
175k
> 4 24.1M 4 1199k 0 0 141k 0 0:02:54 0:00:08 0:02:46
89742
> 4 24.1M 4 1199k 0 0 126k 0 0:03:15 0:00:09 0:03:06
5781
> 4 24.1M 4 1199k 0 0 114k 0 0:03:35 0:00:10 0:03:25
0
> 4 24.1M 4 1199k 0 0 104k 0 0:03:56 0:00:11 0:03:45
0
> 4 24.1M 4 1199k 0 0 98511 0 0:04:17 0:00:12 0:04:05
0
> 4 24.1M 4 1199k 0 0 91188 0 0:04:37 0:00:13 0:04:24
0
> 4 24.1M 4 1199k 0 0 84879 0 0:04:58 0:00:14 0:04:44
0
> .
> . [five hours later we see.... Current Speed is stuck at zero]
> .
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:55 99:56:12
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:56 99:56:11
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:57 99:56:10
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:58 99:56:09
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:03:59 99:56:08
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:00 99:56:07
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:01 99:56:06
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:02 99:56:05
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:03 99:56:04
0
> 4 24.1M 4 1199k 0 0 67 0 4d 09h 5:04:04 99:56:03
0
>
>
>
> --
> ------------------------------------------------------------------------
> # Eric A Lucas
> # ------------
> # "Oh, I have slipped the surly bond of earth
> # and danced the skies on laughter-silvered wings...
> # -- John Gillespie Magee Jr.
> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
>
>
>
--
------------------------------------------------------------------------
# Eric A Lucas
# ------------
# "Oh, I have slipped the surly bond of earth
# and danced the skies on laughter-silvered wings...
# -- John Gillespie Magee Jr.
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug
|
|