Julien Vehent on 15 Aug 2012 04:20:40 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] inotify

On 2012-08-15 06:01, Rich Freeman wrote:
On Tue, Aug 14, 2012 at 11:05 PM, Julien Vehent <julien@linuxwall.info>
During the meeting tonight I mentionned inotify and how it can be used to
keep two directory trees in sync. I used it to keep a local directory tree,
that contained huge amounts of small images, in sync with a bucket in S3
(via s3fs).

An alternative to s3fs is s3cmd.  It has a sync function which
utilizes the fact that s3 maintains an accurate list of hashes to
reduce file transfer.  It does still need to read the local file to
determine the local hash. You could potentially use it in combination
with inotify to reduce local IO, while avoiding copying files if they
haven't actually changed.

One risk of the inotify approach is that if somehow the tress do get
out of sync they'll tend to stay that way, unless you periodically run
a full sync.  That could happen if something is modified while it
isn't running, or if something directly modifies the s3 tree.


Yes. I used to have rsync running every X hours in parallel of inotify, for
this reason. But for the few months I had this running, I don't remember
having a single file out of sync. inotify is very stable and reliable.

- Julien
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug