[PLUG] inotify

[Julien Vehent <julien@linuxwall.info>]

During the meeting tonight I mentionned inotify and how it can be used to 
keep two directory trees in sync. I used it to keep a local directory tree, 
that contained huge amounts of small images, in sync with a bucket in S3 (via 
s3fs).

Here is the script I used: https://gist.github.com/3355264
(see also below)

The inotifywait command listens for create/modify/delete events on the 
content (files and sub directories) of $CONTENTPATH, and if an event is 
triggered, the while loops processes the event.
The event is store in $event. If it is, for example, `MODIFY` the script 
executes:

MODIFY)
    cp -f "$SOURCE" "$DEST"
    type="file"
    ;;

which copies the modified file from the source to the destination.

inotify provides an efficient way to keep two directories in sync in real 
time without the overhead of running rsync, which can be expensive on very 
large trees.

Here is the entire script:

------------------------------------
#!/bin/sh

MONITORLIST="/directory/source/:/directory/destination/ 
/directory/source2/:/directory/destination2/"

for pair in $MONITORLIST; do
    # get the current path
    CONTENTPATH=$(echo $pair|cut -d ':' -f 1)
    S3FSPATH=$(echo $pair|cut -d ':' -f 2)
    CURPATH=$(pwd)

    inotifywait -mr --timefmt '%d/%m/%y %H:%M' --format '%T %w %f %e' \
    -e create -e modify -e delete $CONTENTPATH \
    | while read date time dir file event
    do
        if [[ $dir =~ /\.svn/ ]]
        then
            echo "skipping .svn change"
        else
            ELAPSED_START=$(date "+%s%N")

            FILECHANGE=${dir}${file}
            # convert absolute path to relative
            SOURCE=$(echo "$FILECHANGE" | sed 's_'$CURPATH'/__')

            DEST=$(echo $SOURCE | sed 's_'$CONTENTPATH'_'$S3FSPATH'_')
            type="null"

            case $event in
            CREATE,ISDIR)
                mkdir -p "$DEST"
                type="folder"
                ;;
            MODIFY)
                cp -f "$SOURCE" "$DEST"
                type="file"
                ;;
            DELETE,ISDIR)
                if [ -d $DEST ]; then rmdir "$DEST";fi
                type="folder"
                ;;
            DELETE)
                # delete only if destination exist
                if [ -e $DEST ]; then
                    rm -f "$DEST"
                    type="file"
                else
                    echo "$DEST doesn't exist"
                fi
                ;;
            esac

            ELAPSED_STOP=$(date "+%s%N")
            ELAPSED=$(echo "scale=2; ($ELAPSED_STOP - 
$ELAPSED_START)/1000000"|bc -l)
            if [ "$type" != "null" ]; then
                echo "$date-$time, $type $FILECHANGE was $event at $DEST (in 
$ELAPSED ms)"
            fi
        fi
    done &
done
------------------------------------


--
Julien Vehent - http://jve.linuxwal.info
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug