Julien Vehent on 14 Aug 2012 20:05:14 -0700


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

[PLUG] inotify


During the meeting tonight I mentionned inotify and how it can be used to keep two directory trees in sync. I used it to keep a local directory tree, that contained huge amounts of small images, in sync with a bucket in S3 (via s3fs).

Here is the script I used: https://gist.github.com/3355264
(see also below)

The inotifywait command listens for create/modify/delete events on the content (files and sub directories) of $CONTENTPATH, and if an event is triggered, the while loops processes the event. The event is store in $event. If it is, for example, `MODIFY` the script executes:

MODIFY)
    cp -f "$SOURCE" "$DEST"
    type="file"
    ;;

which copies the modified file from the source to the destination.

inotify provides an efficient way to keep two directories in sync in real time without the overhead of running rsync, which can be expensive on very large trees.

Here is the entire script:

------------------------------------
#!/bin/sh

MONITORLIST="/directory/source/:/directory/destination/ /directory/source2/:/directory/destination2/"

for pair in $MONITORLIST; do
    # get the current path
    CONTENTPATH=$(echo $pair|cut -d ':' -f 1)
    S3FSPATH=$(echo $pair|cut -d ':' -f 2)
    CURPATH=$(pwd)

    inotifywait -mr --timefmt '%d/%m/%y %H:%M' --format '%T %w %f %e' \
    -e create -e modify -e delete $CONTENTPATH \
    | while read date time dir file event
    do
        if [[ $dir =~ /\.svn/ ]]
        then
            echo "skipping .svn change"
        else
            ELAPSED_START=$(date "+%s%N")

            FILECHANGE=${dir}${file}
            # convert absolute path to relative
            SOURCE=$(echo "$FILECHANGE" | sed 's_'$CURPATH'/__')

            DEST=$(echo $SOURCE | sed 's_'$CONTENTPATH'_'$S3FSPATH'_')
            type="null"

            case $event in
            CREATE,ISDIR)
                mkdir -p "$DEST"
                type="folder"
                ;;
            MODIFY)
                cp -f "$SOURCE" "$DEST"
                type="file"
                ;;
            DELETE,ISDIR)
                if [ -d $DEST ]; then rmdir "$DEST";fi
                type="folder"
                ;;
            DELETE)
                # delete only if destination exist
                if [ -e $DEST ]; then
                    rm -f "$DEST"
                    type="file"
                else
                    echo "$DEST doesn't exist"
                fi
                ;;
            esac

            ELAPSED_STOP=$(date "+%s%N")
ELAPSED=$(echo "scale=2; ($ELAPSED_STOP - $ELAPSED_START)/1000000"|bc -l)
            if [ "$type" != "null" ]; then
echo "$date-$time, $type $FILECHANGE was $event at $DEST (in $ELAPSED ms)"
            fi
        fi
    done &
done
------------------------------------


--
Julien Vehent - http://jve.linuxwal.info
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug