Re: [PLUG] check for a file being transmitted via ftp

[JP Vossen <jp@jpsdomain.org>]

On 03/25/2014 02:05 PM, Eric at Lucii.org wrote:
> I expect a file to be routinely transmitted to me via ftp.  A bash script,
> invoked as a cron job, will process and then archive that ftp file.
>
> All of that is easy/routine.
>
> My question is: how do I know that the file is complete?  I don't want to
> start processing the file without being sure that the ftp process is done.

Good one.  I run into this a lot too.  I seem to recall we've talked 
about something similar on this list in the past few months but am too 
lazy to go look now.


> I thought about checking the size of the file in bytes and then comparing that
> to it's size 1 minute earlier... if it's not growing then it's probably done.
> But, that's just *probably* as ftp could be stalled temporarily or worse.

There are at least 2 ways I've thought of this this problem.

1.  Have the remote side send a second tiny 'im_done' flag file.  In a 
way you have the same problem, but since you don't care about the second 
file...  Or if you have specific names:
	fancy_file_1.20140325141934
	fancy_file_1.20140325141934.imdone
	...

2. Check the modified time.  While there are ways to do that using the 
'stat' command with a format so that it's trivial to parse (left to the 
reader), the way I've been using for a while is cheating and using 
'find'.  Either way you still have the race condition--or perhaps 
potential stalled condition is a better name--you mentioned.  You can 
fiddle with FTP timeouts and things but short of the second flag file I 
don't see a way around this.  The variable at least makes it easy to tweak.

----- cut here -----
#VERBOSE='echo'    # For verbose output
VERBOSE=':'        # NoOp for less output
FIND_MINS_OLD='5'  # Make sure file is complete

for incoming_file in $(find /some/path/ -type f -mmin +$FIND_MINS_OLD);
do
    $VERBOSE "Processing $incoming_file..."
    # archive/processing code goes here
done
----- cut here -----

Anyone else have a better way, or ideally a mature daemon that can help?

OK, I just read replies that came in while typing this.  I like the 
'lsof' idea, which I never thought of, which is funny because hey, this 
problem is in the *name* of the command!  :-)

Later,
JP
----------------------------|:::======|-------------------------------
JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
My Account, My Opinions     |=========|      http://www.jpsdomain.org/
----------------------------|=========|-------------------------------
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug