Re: [PLUG] Upload & Append Data to a Server

[JP Vossen <jp@jpsdomain.org>]

On 02/23/2014 07:19 PM, Rich Freeman wrote:
> On Sun, Feb 23, 2014 at 6:02 PM, Casey Bralla <MailList@nerdworld.org> wrote:
> > I've got SFTP access to both machines, and can run a python script to handle
> > the uploads.  What I'd really like is a sftp "append" option, but that doesn't
> > seem to exist.
...
> The usual way of handling this sort of thing is to dump each update
> into a separate file (either random names or timestamped), and upload
> them to wherever they're going, and then have something intelligent on
> the other side parse them.

I agree.  SFTP upload 1 file per minute, or whatever, then 
process/append them on the remote server side *after* upload.

So name it "weather_$(date '+%Y%m%d%H%M%S').dat" or maybe just 
"weather_$(date '+%s').dat" on the Pi side, upload it, then on the 
server side have a script that looks for "weather_*.dat", appends to the 
final data file and deletes.

Depending on how much latency you can tolerate, you can run that 
back-end script once per minute, or every 10 mins, or whatever.

If you really want to get fancy, you can add a check to prevent 
processing a file that is still being written to.  The "hope & pray" 
method would be to run the processing script every 1 minute, but then 
'sleep 15' or something at the top so you *assume* the data file is 
there and not being written.

A less race-condition sensitive way would be to use 'stat' to compare 
file write times over a span of a few seconds, but a much easier way is 
to cheat and use 'find' to do the work for you.  Process files modified 
1 minute ago or longer (needs testing):

for file in $(find /sftp/dump/dir -name 'weather_*.log' -mmin +1); do
    echo "Processing '$file'..."
    # and/or logger -t "$0[$$]" "Processing '$file'"
    cat $file >> /the/real/data/file && rm $file
done


Casey & Timothy, I'd be interested in what you are doing with the Pi, 
just because, and also because I'm thinking about doing a Pi PLUG Preso 
and could use ammo.  Are you guys working from plans on the 'Net or 
winging it?  Clues?

Later,
JP
----------------------------|:::======|-------------------------------
JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
My Account, My Opinions     |=========|      http://www.jpsdomain.org/
----------------------------|=========|-------------------------------
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug