Skip to Content

Running rsync on Windows

Posted on

Pelican is a static blog generator, so it runs and creates a load of plain old .html files which sit on your local drive. It does have several methods to push your content to your server (or you can host on github):

[D:\Dropbox\pelican]make
'Makefile for a pelican Web site                                        '
'                                                                       '
'Usage:                                                                 '
'   ssh_upload                       upload the web site via SSH        '
'   rsync_upload                     upload the web site via rsync+ssh  '
'   dropbox_upload                   upload the web site via Dropbox    '
'   ftp_upload                       upload the web site via FTP        '
'   github                           upload the web site via gh-pages   '

FTP isn’t the most secure thing in the world so I skipped to ssh_upload. It worked but it uploads every file, every time. Since I usually only add one blog post at a time and all the other files remain unchanged but still had to be copied to the server this was not very optimal.

Yuck.

Rsync is a great utility that can copy files and directories between destinations. What makes rsync special is it can only copy the differences! So if I add a new blog post, rsync will scan my local directory, my remote directory, figure out 1 added a file and only copy that file.

SSH uploads would take several minutes while rsync takes several seconds.

Unfortunately rsync isn’t native on Windows but after some more Googling I found cwRsync. They offer a free and paid version. To make sure it worked I just downloaded the free version. You unzip, run the installer and end up with a C:\Program Files (x86)\cwRsync directory. Looking in there you’ll see some cygwin files. Don’t panic. cwRsync is much lighter than a full cygwin install!

I then tried my Pelican rsync_upload script. It ran but was throwing permission errors:

Permission denied, falling back on umask rsync: mkstemp “/thecrumb/posts/…index.html.SYo96q” failed: Permission denied (13)

[ Insert rant about how crappy Windows is here ]

Back to Google. Turns out you need to set an option for rsync. Normally you’d edit your fstab file on Linux but on Windows you need to create one:

C:\Program Files (x86)\cwRsync\etc\fstab (no file extension!)

And within that put the following:

# In this example, my cwRsync dir is located at: "C:/Program Files (x86)/cwRsync"
# Filename: "C:/Program Files (x86)/cwRsync/etc/fstab"
none /cygdrive cygdrive binary,posix=0,user,noacl 0 0

I’m not sure what this does exactly but I believe it simply tells rsync to ignore file permissions.

Once I did that I retried rsync and we had success!

Next steps - adding a key pair so I can run rsync without a password and then I can automate the entire process.