Fred Stluka on 21 Dec 2013 11:43:10 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Rsync advice wanted


Michael,

Sorry for the delay.  Been busy...

Yes, this sounds reasonable.  Rsync is very good at keeping 2
directories in sync, so you should be able to set it up to do
exactly what you need.

You're using --remove-source-files to remove files from the
source after copying them to the target.  That gives you an
empty source tree most of the time, indicating that they have
all been successfully copied to the target tree.

Do you give the receiving vendor a way to delete them from
target after STFPing them out?  If so, do you want a way to
keep copies in source until the vendor has deleted them from
target?  You can do that via the --delete option when rsyncing
in the opposite direction.

See my rsync tips for more ideas:
- http://bristle.com/Tips/Unix.htm#rsync

--Fred
------------------------------------------------------------------------
Fred Stluka -- mailto:fred@bristle.com -- http://bristle.com/~fred/
Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
Open Source: Without walls and fences, we need no Windows or Gates.
------------------------------------------------------------------------

On 12/16/13 1:12 PM, Michael Leone wrote:
I have a setup now where I have a RH server in my DMZ, and my vendors
use SFTP to copy files to us. I also have a Ubuntu VM on my trusted
LAn that will reach into that box (via SSH; the DMZ has copies of it's
keys); run a script to sweep all new files into a single ZIP file;
rsync that back into the trusted LAN; and deletes the source on the
DMZ. I had a whole lot of help from this list with that script close
to 3.5 years ago now (thanks again!) and it's been running fine.

Now, I have a need to go in the opposite direction. We want to send
files to the DMZ, for other vendors to pick up (also over SFTP).
Ideally, Here's what I am thinking, please comment.

These new vendors will have home directories in an entirely different
tree from the vendors that send in files. Reason: I can then easily
duplicate this tree on the trusted side (we're talking less than 10
user directories, total).

When we have a file to send to User-1, we write it in the "home
directory" of User-1 on the trusted side. Then, on a schedule, a
script will make an rsync connection over SSH to the DMZ; replicate
any files on the trusted side to the DMZ side (ignoring any files on
the DMZ side that happen to already exist), and then deleting the file
from the trusted side.

Rsync should be able to do this, with these options?

rsync -rvz  --ignore-existing --remove-source-files
<trusted-side-tree>  <DMZ-user@DMZ-IP:/DMZ-side-tree>

DMZ-user can be given rights into the user home directory, so it can
write files there.

This way, the script cleans out the trusted side (the "outgoing
files"), while not touching any files that the vendor may have not yet
SFTPed out.

Thoughts? I know I've missed something, or made some stupid assumption.

I have very very little experience writing Linux scripts, but have
written many on Windows (CMD, Powershell) so it's more the specific
implementation on Linux that I am weak on.
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug


___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug