JP Vossen on 23 Feb 2014 16:44:28 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Upload & Append Data to a Server


On 02/23/2014 07:19 PM, Rich Freeman wrote:
On Sun, Feb 23, 2014 at 6:02 PM, Casey Bralla <MailList@nerdworld.org> wrote:
I've got SFTP access to both machines, and can run a python script to handle
the uploads.  What I'd really like is a sftp "append" option, but that doesn't
seem to exist.
...
The usual way of handling this sort of thing is to dump each update
into a separate file (either random names or timestamped), and upload
them to wherever they're going, and then have something intelligent on
the other side parse them.

I agree. SFTP upload 1 file per minute, or whatever, then process/append them on the remote server side *after* upload.

So name it "weather_$(date '+%Y%m%d%H%M%S').dat" or maybe just "weather_$(date '+%s').dat" on the Pi side, upload it, then on the server side have a script that looks for "weather_*.dat", appends to the final data file and deletes.

Depending on how much latency you can tolerate, you can run that back-end script once per minute, or every 10 mins, or whatever.

If you really want to get fancy, you can add a check to prevent processing a file that is still being written to. The "hope & pray" method would be to run the processing script every 1 minute, but then 'sleep 15' or something at the top so you *assume* the data file is there and not being written.

A less race-condition sensitive way would be to use 'stat' to compare file write times over a span of a few seconds, but a much easier way is to cheat and use 'find' to do the work for you. Process files modified 1 minute ago or longer (needs testing):

for file in $(find /sftp/dump/dir -name 'weather_*.log' -mmin +1); do
    echo "Processing '$file'..."
    # and/or logger -t "$0[$$]" "Processing '$file'"
    cat $file >> /the/real/data/file && rm $file
done


Casey & Timothy, I'd be interested in what you are doing with the Pi, just because, and also because I'm thinking about doing a Pi PLUG Preso and could use ammo. Are you guys working from plans on the 'Net or winging it? Clues?

Later,
JP
----------------------------|:::======|-------------------------------
JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
My Account, My Opinions     |=========|      http://www.jpsdomain.org/
----------------------------|=========|-------------------------------
"Microsoft Tax" = the additional hardware & yearly fees for the add-on
software required to protect Windows from its own poorly designed and
implemented self, while the overhead incidentally flattens Moore's Law.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug