Large File Performance: Amazon EFS vs ObjectiveFS

For processing large files such as videos, images and life science data, sequential read and write performance is a key factor when selecting a filesystem. We compared the performance of two shared filesystems commonly used to scale these workloads on AWS EC2 instances: Amazon Elastic File System (EFS) and ObjectiveFS. For details on how Amazon EFS differs from ObjectiveFS, see here.

Summary

ObjectiveFS has 670+ MB/s throughput per node for large sequential writes and is over 6.5X faster than Amazon EFS.

ObjectiveFS has 550+ MB/s throughput per node for large sequential reads and is around 5X faster than Amazon EFS.

Test Setup

We selected an EC2 instance type with 10 Gbit network so the performance test is not limited by network bandwidth. We compared the performance of an EFS filesystem with bursting throughput, an EFS filesystem with provisioned throughput and an ObjectiveFS filesystem, all running on a r7a.8xlarge EC2 instance with Ubuntu 22.04 in the us-east-2 region.

For both EFS with bursting and provisioned throughput, we used the default General Purpose performance mode since this test was done from a single machine. For the EFS filesystem with provisioned throughput, we selected 50 MiB/s for throughput based on cost (US$300/month) which is similar to an ObjectiveFS Team plan.

For ObjectiveFS, we used ObjectiveFS version 7.1, enabled enhanced multithreading for best throughput and used the default memory cache setting.

Note: For EFS, we kept the test within the burst time limit to get the best possible EFS performance. For a 100 GiB filesystem, the EFS burst limit is 72 minutes per day. After that, Amazon EFS performance is limited to 5 MiBps for writes and 15 MiBps for reads. For more details, see the Throughput Scaling section of Amazon EFS Performance. ObjectiveFS does not limit performance and can sustain the full throughput consistently 24x7.

Setup Summary

Test

1. Large Sequential Write

The write test consists of writing a large non-compressible 10 GB file to the filesystem. The source file was pregenerated using shred on the local NVMe SSD and is cached in the kernel cache to ensure reading the source file is not the bottleneck during the write test.

# touch /nvme/10G_File
# shred -n1 --size=10G /nvme/10G_File

# # Warm source file in kernel cache for faster reading
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null
# dd if=/nvme/10G_File of=/dev/null bs=1M &> /dev/null

# time dd if=/nvme/10G_File of=$TESTDIR/10G_File bs=1M

2. Large Sequential Read

The read test consists of reading a large non-compressible 10 GB file from the filesystem. For ObjectiveFS, we measured the throughput for the initial read from S3 (cold cache) and subsequent read when the data is cached in memory. Kernel caches are dropped using echo 3 > /proc/sys/vm/drop_caches before each run for all cases (except for the warm memory cache case) to avoid any effects from kernel caching.

# echo 3 > /proc/sys/vm/drop_caches
# time dd if=$TESTDIR/10G_File of=/dev/null bs=1M

Results

1. Large Sequential Write Performance

The table below shows the throughput for writing a non-compressible 10 GB file to EFS and ObjectiveFS. The EFS filesystem with bursting throughput reached 104 MB/s (see note a), while the EFS filesystem with provisioned throughput stayed at the provisioned 52 MB/s. ObjectiveFS achieved 676 MB/s and is over 6.5X faster than the faster of the two EFS filesystems.

a: The EFS burst performance is limited to 72 minutes per day for a 100GiB filesystem. Subsequent throughput is limited to 5 MiBps for writes (see “Throughput Scaling” in Amazon EFS Documentation).

2. Large Sequential Read Performance

The table below shows the throughput of reading a non-compressible 10 GB file from EFS and ObjectiveFS. The EFS filesystem with bursting throughput was 109 MB/s (see note a), while the EFS filesystem with provisioned throughput stayed at the provisioned amount of 53 MB/s. ObjectiveFS achieved 556 MB/s for large sequential reads directly from S3 and 12.4 GB/s when the data is cached in the memory cache.

a: The EFS burst performance is limited to 72 minutes per day for a 100GiB filesystem. Subsequent throughput is limited to 15 MiBps for reads (see “Throughput Scaling” in Amazon EFS Documentation).

Conclusion

For large sequential writes directly to S3, ObjectiveFS writes at over 670 MB/s per node and is 6.5X faster than EFS. For large sequential reads directly from S3, ObjectiveFS reads at around 550 MB/s and is over 5X faster than EFS. For large reads that can be cached in the memory cache, we see a throughput of 12.4 GB/s (around 115X faster than EFS).

To see how small file workloads compare, see how ObjectiveFS compared to EFS on small file workloads. To see how ObjectiveFS performs for your workload, you can download a copy and try it with our free trial.

Last updated by ObjectiveFS staff, Sep 21, 2023
ObjectiveFS is a high performance scalable shared file system for Linux and macOS that scales automatically and gives you unlimited storage. If you have questions or article idea suggestions, please email us at support@objectivefs.com