Sign in Subscribe

rsync on macOS: Matters Arising

Saïd

• 4 min read
rsync on macOS: Matters Arising

I recently had to transfer large files on Windows to macOS Catalina (10.15.7) using rsync, but I met a few snags on the way. I already wrote about my experience using rsync for the first time on Windows.

This article is the concluding part that delves into the issues I encountered on the macOS side of the transfer.

Too many bytes?

The transfer went smoothly for the most part, until rsync got to an ~18GB .vdi file used by one of my VirtualBox VMs:

bash -c "rsync -avzh -P --remove-source-files --stats --timeout=60  VirtualBox 'mac@10.0.0.104:/Users/mac/Windows8-Lenovo/'"
(mac@10.0.0.104) Password:
building file list ...
65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.63G  54%   68.12MB/s    0:01:55
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(453) [generator=2.6.9]
rsync error: error in rsync protocol data stream (code 12) at io.c(823) [sender=3.2.3]

I scanned the error message and figured it might have been due to a minor network issue, so I re-attempted the transfer two more times and the error was fairly reproducible.

Here's the output from the 2nd try:

65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.66G  54%   10.34MB/s    0:12:37
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

Here's the output from the 3rd try:

65 files to consider
VirtualBox/Ubuntu 18.04 LTS + Upwork/Ubuntu 18.04 LTS.vdi
          9.62G  54%   68.44MB/s    0:01:54
rsync: [sender] write error: Broken pipe (32)
rsync error: timeout in data send/receive (code 30) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]
rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

So, after 3 attempts, the transfer fails at exactly 54% of the way during the transfer of an 18GB file.

Filesystem limitation?

I've experienced a similar kind of error a long time ago while transferring a large file to FAT-based filesystem used by Windows, so my first thought was that this was also a filesystem limitation issue.

The difference here was that Windows was the source and not the destination of the large file transfer, so I decided to search for the first error message in the output "rsync: [sender] write error: Broken pipe (32)".

Unable to rsync due to broken pipe

The first result on Google: Unable to rsync due to broken pipe turned out to be especially useful in pointing me towards the underlying issue–it turned out I was using two different versions of rsync.

Just to be sure, I went ahead to re-read the error output much more closely:

rsync: [sender] write error: Broken pipe (32)

rsync error: timeout in data send/receive (code 30) at /AppleInternal\

/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/\

rsync-54.120.1/rsync/io.c(164) [receiver=2.6.9]

rsync: connection unexpectedly closed (1747 bytes received so far) [generator]

rsync error: error in rsync protocol data stream (code 12) at \

/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/\

Sources/rsync/rsync-54.120.1/rsync/io.c(453) [generator=2.6.9]

rsync error: error in rsync protocol data\

stream (code 12) at io.c(823) [sender=3.2.3]

On the sending machine (Windows) I had rsync v3.2.3, while on the receiving machine (macOS) I had rsync v2.6.9 installed:

rsync --version
rsync  version 2.6.9  protocol version 29
Copyright (C) 1996-2006 by Andrew Tridgell, Wayne Davison, and others.
<http://rsync.samba.org/>
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles,
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Fixing the problem

The fix was fairly easy–I had to update the dated v2.6.9 copy of rsync on macOS to match v3.2.3 on Windows.

brew install rsync

There was a slight problem though, because of how my Homebrew installation was setup to put all binaries under ~/homebrew/bin.

Version 2.6.9 of rsync at /usr/bin/rsync was getting found first in my PATH instead of v3.2.3 that I had just installed to ~/homebrew/bin/rsync.

Fixing a latent problem

I had been putting off the Homebrew PATH issue for a while due to my unfamiliarity with the default zsh on macOS, but this time, I decided I was going to properly research how zsh differs from the more familiar bash, so I could fix the PATH issue the correct way, rather than cargo-cult magical incantations from the Internet.

I've distilled my research into two comment lines just before the actual fix for zsh:

# In https://apple.stackexchange.com/a/425894 it mentions /etc/zprofile (different from my pre-existing ~/.zshenv and ~/.zshrc)

# In /etc/zprofile it says to add customizations to ~/.zprofile instead, so this is where our Homebrew's rsync v3.2.3 can overshadow Apple's v2.6.9 in the PATH

echo "export PATH=/Users/mac/homebrew/bin:$PATH" > ~/.zprofile

Running rsync --version from anywhere now gives the newer version:

rsync  version 3.2.3  protocol version 31
Copyright (C) 1996-2020 by Andrew Tridgell, Wayne Davison, and others.
Web site: https://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, hardlink-specials, symlinks, IPv6, no atimes,
    batchfiles, inplace, append, ACLs, xattrs, optional protect-args, iconv,
    symtimes, no prealloc, stop-at, crtimes, file-flags
Optimizations:
    SIMD, asm, openssl-crypto
Checksum list:
    xxh128 xxh3 xxh64 (xxhash) md5 md4 none
Compress list:
    zstd lz4 zlibx zlib none

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

Sign up for more like this.

Enter your email
Subscribe