Sync It, Baby!

I wrote before on how most hard drives I buy turn out to be allergic to my data. I do not know what it is about my data that pisses their platters off and spins them crazy, but it so often happens that I can hardly keep my sanity.

Even though I use an additional hard drive to mirror my data on a nightly basis, this to me is still is not enough guarantee for the safety of my precious data. Just as the first drive could blow up, it could happen that the second would follow suit before I notice the disaster. Without any doubt, there is no way in which one can totally safeguard his data, but the more levels of protection one could employ, the less tears he might later shed.

Simple Storage Service, or S3, from Amazon.com caught my attention on its day of release due to its relatively low cost compared to similar, off-site storage services like rsync.net. Rather than going with a plan-based billing style as employed by many services, Amazon charges for one’s monthly usage of bandwidth and storage, per megabyte, and approximated to the nearest US cent. With S3, one gigabyte of bandwidth would cost you $0.20, while each gigabyte of monthly storage set you back $0.15. Even more interesting than these numbers is how they calculate them. For storage, they actually charge you for your disk consumption per hour. So if you uploaded 1GB of data in the last hour of the last day of the month, you will only be charged for that hour’s cost, a mere $0.0002! If you uploaded 10MB worth of files in the first hour of the first day of the month, and nothing more throughout the month, then the month’s storage cost would be only $0.0015.

With these numbers in mind, I still cannot make backups of all my data. Not only it will be terribly expensive to backup hundreds of gigabytes per month, but I also cannot push this amount of data through my poor 128Kbps upstream without clogging up my tubes. Rather, what I plan to do is to backup only the most important data that would truly break my heart and tear my soul if I were to lose them. Such data would hardly be hundreds of megabytes, a few gigabytes tops. And with incremental rather than full backups, I won’t need to push much data out anyway each month.

From there I went on and gave it a shot. Only a few months after the service has started and there were already several open-source projects that offered various implementations of the S3 API. I picked s3sync, a Ruby script interface to S3 that behaves somewhat similar to rsync in its synchronization of files and directories. I got the script installed on my NSLU2 and on my VPS webserver, set then set weekly cron jobs for syncing.

After one month of using S3, of the interesting features that I came to use are the ability to send the data encrypted over SSL, which s3sync gladly supports; that it retains the UNIX permissions and ownership of files and directories; and that it allows me to set Access Control on files, which could allow, if I want, to share some of my data publicly in directories (Amazon’s technical term is buckets) as URLs like http://s3.amazonaws.com/a-bucket-name-not-there/. An extremely interesting feature that I did not came to use, but only tried out, was their Bittorrent support! Just add the suffix “?torrent” to any publicly shared file, and voila, you get a torrent for that file tracked by Amazon. The owner of the S3 account does not need to set up or enable anything. All files publicly shared can be downloaded through HTTP, or through Bittorrent by simply adding the said suffix.

My bill for the first month came with 0.588 GB of data transfered and 0.52 GB of disk storage cost me a staggering $0.12 and $0.08, respectively, for a total of 20 cents!! Their “Billing Statement” e-mail was kind of funny to read, since I knew they may be paying more than those 20 cents to process my credit card.

This e-mail confirms that your latest billing statement is available on the
AWS web site. Your account will be charged the following:

Subtotal: $0.20 (plus applicable taxes)

And in case you were wondering, no taxes were applicable!

Posted February 12th, 2007. Filed under: Posts.