Commvault Announced Acquisition of Clumio

// 22 Aug 2023

A closer look at speed and scale results when backing up Amazon S3

Woon Jung, Co-Founder & CTO
ShareTwitterfacebookLinkedin

Backing up Amazon S3 data

As millions of organizations adopt the simplicity of Amazon S3 to store and manage business-critical data, the need for high-performance backup and recovery is becoming increasingly clear. The applications, data lakes, and machine learning models that run on Amazon S3 are often powered by critical and sensitive data that needs to be backed up on a continuous basis, and retrievable near-instantly. Clumio has pioneered backup and recovery for Amazon S3 (we even wrote a book on it) for 2 years now, and it’s exciting to see other players join the market.

Using AWS Backup for S3

AWS Backup recently announced a performance improvement for S3 that speeds up their backup times for buckets containing over 300 million objects. The improvement additionally allows backup of buckets containing more than their previous upper limit of 3 billion objects. Below is the average expected performance with these improvements as published in the AWS Backup Developer Guide, and we have added a column calculating the objects per second figure for each.The published results of AWS Backup services notably lacked restore performance metrics.

AWS Backup

AWS Backup

As published in Amazon S3 Backup Developer Guide

Number of objects Bucket size Estimated time to complete initial backup Objects per second (calculated by Clumio)
135 million 425 GB 31 hours 1,210
670 million 800 TB 38 hours 4,898
5 billion 6 PB 100 hours 13,889
7.5 billion 370 TB 180 hours 11,574

Clumio’s performance results

We thought this might be a good opportunity to share some results of Amazon S3 backups with Clumio. Architected as a serverless data processing pipeline, speed and scale of Amazon S3 backups have been the primary focus at Clumio since we started supporting it in 2021.

Clumio

Clumio

Number of objects Bucket size Actual time to complete initial backup Objects per second
100 million 5 TB 80 minutes 20,833
3.38 billion 1.54 PB 53 hours 17,715
6.14 billion 4.24 PB 95 hours 17,953
24.4 billion 26.7 PB 231 hours 29,341
24.69 billion 759 TB 184 hours 37,274

What makes a backup fast

As you can see, object count and overall data volume both play a role in the backup time of a given bucket. This is illustrated by the two examples of buckets containing about 24 billion objects, but at very different sizes, resulting in very different backup times. Generally speaking, the object count should be your first consideration in estimating backup time, with data volume following. While other factors will always influence backup speed, you can count on Clumio’s backup speed being relatively predictable from the highest to the lowest object counts and data volumes. Notice that Clumio’s performance does not reduce for the lowest object count example, and also scales to well above 20 billion objects per bucket, resulting in exceptional backup speed regardless of the customer’s S3 environment. Indeed, we have customers backing up buckets that are a few GB to tens of PB.

Enabling high performance backups without affecting application performance

In addition to being fast, Clumio also detects when buckets being backed up are in demand, and throttles the backup processing to allow the primary application to perform optimally even while backup is in process. This is especially important during large seeding backups, which can get large enough to take up significant resources and API quotas. This means if an application wasn’t making any calls to a given bucket, its backup time could be as much as 50% faster. However, that’s not a typical scenario. Many applications, especially those with a global user base, need to run continuously. It’s important to preserve their functionality, especially for large buckets whose initial backups will take several days.

What about restoring data?

While backup speed is an important benchmark, restore speed is even more important. After all, the point of backups is to restore data quickly when you need it most. Even though the AWS Backup performance results didn’t showcase their restore metrics, let’s take a look at the restore speeds that you can expect with Clumio for Amazon S3 data.

Clumio

Clumio

Number of objects Bucket size Actual time to complete restore Objects per second
1.6 million 236 MB 13 minutes 2051
1.1 million 1.03 TB 29 minutes 632
22.15 million 2.64 TB 2h 37m 2351
24 million 410 GB 2h 44m 2439
30 million 34 TB 56 minutes 8929

S3 restores most commonly go to new buckets, which start with lower API rate limits. AWS scales up the API limits automatically as more objects are created in the destination bucket. This delayed scaling impacts the rate at which objects are created in the destination bucket during the initial phases when API limits are low. The restore process can be accelerated by working with AWS to proactively increase the limits before starting.

Achieving breakthrough zero-RTO restore performance with Clumio Instant Access

For large S3 buckets, waiting for a full restore of your backup to resume operations might be too disruptive. On the other hand, maintaining a warm failover (such as a replicated bucket) of your data all the time can be expensive (and doesn’t offer point in time recovery, air gap, and other features of backups). Clumio’s Instant Access feature exposes an S3 endpoint for applications to directly access data from backups without having to restore the objects at all. This reduces the RTO to minutes, even for buckets containing tens of petabytes.

Your keys to fast and performant S3 backups

Hopefully this has been a helpful exploration of backup and restore speeds for Amazon S3. When assessing a backup solution for Amazon S3, be sure to test restore performance! If you’d like to learn how Clumio’s performance can help you better manage your data estate, request a personalized demo.

About the author

Woon Jung is the Chief Technology Officer and co-founder at Clumio. Woon has 15 years of technology engineering experience, building core operating system technologies and cloud services. He comes to Clumio from Nutanix, where he was a Senior Staff Engineer. Prior to that, he was a Founding and Distinguished Engineer at PernixData.