Arq Backup—Too Many, Too Fast

Too many SFTP connections, too fast that is.

After configuring an SFTP destination, Arq 5.10 opens and closes multiple SFTP connections per second while backing up. This causes my web hosting service, 1and1.com, to treat Arq as if it’s doing a Denial of Service attack (which it sort of is), and blocks further connections resulting in most backups failing. I get maybe one backup a month that makes it through.

It was working when I first bought Arq 5.0, then it was broken, I reported the problem, it was fixed, then broken again. After my last report support replied in July,

I’m afraid Arq is going to need multiple SFTP connections.

So the developer refuses to fix the problem. I can’t remember the number of file transfer apps I’ve used over the years to access 1and1’s servers without an issue. Arq is the only app that will not work, and it’s design intent.

This post was prompted by a tweet about this very issue. So it’s not just causing problems for me:

In addition to my 1and1 backup, I also back up to a Synology server and have verified the rapid connection open and closes, although when thousands of errors are reported, that is usually fixed by clearing the cache. (Aside: Why does Arq feel the need to report thousands of the same errors? It should report a few and then throttle the remaining errors and say, “300,000 similar errors.” Seriously, who’s going to look at 300,000 error messages? And why do I need to clear the cache to stop these errors? Arq shouldn’t let the situation deteriorate to this point. But I digress.)

Arq’s design of rapidly and unnecessarily opening and closing SFTP connections just to do a backup is poor design. There’s no sugar-coating it. There is not another single program I can ever remember using that transfers large numbers or sizes of files to a remote server that feels it necessary to behave this way. While some parallelism may be good, decent apps will allow you to control the maximum number of simultaneous connections. And they sure as hell don’t open and close them multiple times a second. That’s just silly and awful design. Well designed apps open the connection once per session and close it when all commands have been executed. Needlessly opening and closing connections slows the backup and puts unnecessary load on the client and server to set up and tear down the connections.

In the late 1980’s when I was implementing the TCP/IP family of protocols on Nortel’s big DMS switches, I remember reading about an interoperability rule, “Be conservative in what you send, be liberal in what you accept.” Arq violates that rule by rapidly opening and closing simultaneous connections. Arq is clearly not conservative enough, but is 1and1 not liberal enough? 1and1 needs to protect its shared servers from hostile apps like Arq. I think 1and1’s throttling setting is fine since I’ve had no issues with any other app. The full blame for this lack of interoperability falls squarely on Arq’s shoulders.

This is not the only design issue I have with Arq but I’ve learned not to expect change. If it’s broken now, it will remain broken and I will just need to live with it. So I don’t rely on the 1and1 backup to be the only backup. It works so seldom that it’s really only a quaternary backup at best.

Author: Tom

Destroyer of software. If I haven't tested it, it hasn't been tested.

One thought on “Arq Backup—Too Many, Too Fast”

Leave a Reply

Your email address will not be published. Required fields are marked *