JP Vossen on 25 Mar 2014 11:23:04 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] check for a file being transmitted via ftp

On 03/25/2014 02:05 PM, Eric at wrote:
I expect a file to be routinely transmitted to me via ftp.  A bash script,
invoked as a cron job, will process and then archive that ftp file.

All of that is easy/routine.

My question is: how do I know that the file is complete?  I don't want to
start processing the file without being sure that the ftp process is done.

Good one. I run into this a lot too. I seem to recall we've talked about something similar on this list in the past few months but am too lazy to go look now.

I thought about checking the size of the file in bytes and then comparing that
to it's size 1 minute earlier... if it's not growing then it's probably done.
But, that's just *probably* as ftp could be stalled temporarily or worse.

There are at least 2 ways I've thought of this this problem.

1. Have the remote side send a second tiny 'im_done' flag file. In a way you have the same problem, but since you don't care about the second file... Or if you have specific names:

2. Check the modified time. While there are ways to do that using the 'stat' command with a format so that it's trivial to parse (left to the reader), the way I've been using for a while is cheating and using 'find'. Either way you still have the race condition--or perhaps potential stalled condition is a better name--you mentioned. You can fiddle with FTP timeouts and things but short of the second flag file I don't see a way around this. The variable at least makes it easy to tweak.

----- cut here -----
#VERBOSE='echo'    # For verbose output
VERBOSE=':'        # NoOp for less output
FIND_MINS_OLD='5'  # Make sure file is complete

for incoming_file in $(find /some/path/ -type f -mmin +$FIND_MINS_OLD);
    $VERBOSE "Processing $incoming_file..."
    # archive/processing code goes here
----- cut here -----

Anyone else have a better way, or ideally a mature daemon that can help?

OK, I just read replies that came in while typing this. I like the 'lsof' idea, which I never thought of, which is funny because hey, this problem is in the *name* of the command! :-)

JP Vossen, CISSP            |:::======|
My Account, My Opinions     |=========|
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
Philadelphia Linux Users Group         --
Announcements -
General Discussion  --