Re: [PLUG] Rsync advice wanted

[Michael Leone <turgon@mike-leone.com>]

On Sat, Dec 21, 2013 at 2:43 PM, Fred Stluka <fred@bristle.com> wrote:
>
> Michael,
>
> Sorry for the delay.  Been busy...
>
> Yes, this sounds reasonable.  Rsync is very good at keeping 2
> directories in sync, so you should be able to set it up to do
> exactly what you need.
>
> You're using --remove-source-files to remove files from the
> source after copying them to the target.  That gives you an
> empty source tree most of the time, indicating that they have
> all been successfully copied to the target tree.

Yep, that's what I would want.

> Do you give the receiving vendor a way to delete them from
> target after STFPing them out?  If so, do you want a way to
> keep copies in source until the vendor has deleted them from
> target?  You can do that via the --delete option when rsyncing
> in the opposite direction.

Well, I plan to make a ZIP file (with directory structure) of all the
outgoing files before rsyncing, and copying the ZIP to a separate
archive folder, as a backup.

> See my rsync tips for more ideas:
> - http://bristle.com/Tips/Unix.htm#rsync

I will do that, thanks!

>
> --Fred
> ------------------------------------------------------------------------
> Fred Stluka -- mailto:fred@bristle.com -- http://bristle.com/~fred/
> Bristle Software, Inc -- http://bristle.com -- Glad to be of service!
> Open Source: Without walls and fences, we need no Windows or Gates.
> ------------------------------------------------------------------------
>
>
> On 12/16/13 1:12 PM, Michael Leone wrote:
>>
>> I have a setup now where I have a RH server in my DMZ, and my vendors
>> use SFTP to copy files to us. I also have a Ubuntu VM on my trusted
>> LAn that will reach into that box (via SSH; the DMZ has copies of it's
>> keys); run a script to sweep all new files into a single ZIP file;
>> rsync that back into the trusted LAN; and deletes the source on the
>> DMZ. I had a whole lot of help from this list with that script close
>> to 3.5 years ago now (thanks again!) and it's been running fine.
>>
>> Now, I have a need to go in the opposite direction. We want to send
>> files to the DMZ, for other vendors to pick up (also over SFTP).
>> Ideally, Here's what I am thinking, please comment.
>>
>> These new vendors will have home directories in an entirely different
>> tree from the vendors that send in files. Reason: I can then easily
>> duplicate this tree on the trusted side (we're talking less than 10
>> user directories, total).
>>
>> When we have a file to send to User-1, we write it in the "home
>> directory" of User-1 on the trusted side. Then, on a schedule, a
>> script will make an rsync connection over SSH to the DMZ; replicate
>> any files on the trusted side to the DMZ side (ignoring any files on
>> the DMZ side that happen to already exist), and then deleting the file
>> from the trusted side.
>>
>> Rsync should be able to do this, with these options?
>>
>> rsync -rvz  --ignore-existing --remove-source-files
>> <trusted-side-tree>  <DMZ-user@DMZ-IP:/DMZ-side-tree>
>>
>> DMZ-user can be given rights into the user home directory, so it can
>> write files there.
>>
>> This way, the script cleans out the trusted side (the "outgoing
>> files"), while not touching any files that the vendor may have not yet
>> SFTPed out.
>>
>> Thoughts? I know I've missed something, or made some stupid assumption.
>>
>> I have very very little experience writing Linux scripts, but have
>> written many on Windows (CMD, Powershell) so it's more the specific
>> implementation on Linux that I am weak on.
>> ___________________________________________________________________________
>> Philadelphia Linux Users Group         --        http://www.phillylinux.org
>> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
>> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
>>
>>
> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug