Test Driving AWS Elastic File System (EFS)

The new ElasticFilesystem (EFS) service from AWS presents customers with an attractive low cost shared (NFS v4) storage solution that is highly durable and scalable. In this post we aim to put EFS through its paces in terms of both performance and reliability.

AWS has a knack for releasing new services and product updates at breakneck speed. 2015 was no exception and saw the release of a number of services that high end customers had been enthusiastically requesting for quite some time. To name but a few of these, VPC EndPoints, VPC FlowLogs, AWS WAF and the new NAT Gateway service are all welcome additions to the AWS arsenal. Whilst these services are now in general release, another service, ElasticFilesystem (EFS), was announced back in April 2015 but is still in preview mode. Cloudten has been lucky enough to get early access to trial this service as we have a number of customers who are keen to try it.

So what’s so appealing about EFS ? AWS prides itself on the range of storage services available, from Glacier and S3 all the way up to high end PIOPS EBS volumes. Up until this point, however, there has been no service to implement a shared filesystem. It’s important to remember that S3 is a filestore, not a filesystem. It’s a great option to use S3 API calls to reference static objects, however, many people have a legitimate need to be able to access a proper shared filesystem (eg NFS) with journalling and (near) immediate consistency capabilities. EFS aims to fill that gap. For anyone even considering S3FS as an option, here is a post we put together last year heavily advising against it.

Customers have been using a number of alternative solutions to implement an NFS style share. One popular open source method is to use GlusterFS, which is itself an implementation of FUSE. We have used this ourselves, very successfully, for a number of our clients.

There are also a number of commercial third party products which offer shared storage services in the AWS Marketplace. Vendors such as Zedara and NetApp both provide shared storage resources from infrastructure hosted inside Direct Connect enabled facilities (i.e. not actually in an AWS datacentre but at a location that has a very high speed pipe into one). It already seems that Zadara are trying to diffuse the potential impact of EFS on their AWS business.

Figure 1. A high level diagram showing the basic premise of GlusterFS. For more information, see one our previous blog posts.

Figure 2. A high level representation of a third party vendor storage solution hosted at a DirectConnect facility.

 

The purpose of this post is to present an overview of our findings in relation to EFS performance and usability. It should also be noted that EFS is officially in Preview and it is our understanding that changes are still being made to it prior to general release. There are also a list of (quite reasonable) resource limits defined on the AWS site which may change prior to general release.

From an initial reading of the available documentation around EFS, there are a number of things that we found interesting:

  • You don’t have the option of setting the speed or PIOPS of the EFS volume. AWS are notoriously secretive about releasing the details of the underlying workings of their services. It seems that EFS is somewhat similar to an Elastic Load Balancer (ELB) in this respect in that AWS want to take care of any horizontal or vertical scaling under the covers.
  • You don’t have the option of defining the size of the EFS volume you want to create. Similar to an S3 bucket where you simply pay for the storage you use, it looks like EFS offers ‘limitless’ storage. For trivia buffs, when you mount an EFS partition on an EC2 instance, the size shows up as 9007199254740992 KB (i.e. quite big!).
  • The cost. The service is only currently in preview in the US-West (Oregon) region and comes in at USD $0.30 per GB per month. This may seem expensive when compared to standard EBS volumes (a standard SSD comes in at about $0.10 per GB/m and magnetic is cheaper still) but you’re actually getting a lot more than that. We will delve more into this later.
  • Redundancy. AWS state that regardless of how many AZs you create a network share across, your data is automatically replicated over multiple AZs for increased durability. The service was originally only planned for general release across the three largest AWS regions (us-west-2:Oregon,us-east-1:North Virginia,eu-west-1:Ireland). This implies that this could be one of those services, such as RDS Aurora, that requires three live AZs in a region to satisfy a durability SLA. However, fear not Australia, as there could be some very positive news coming from AWS around Q1/2 2016 on this front.

Getting Started

When you create an EFS volume you can specify one or more AZs to span across. Once the shared volume is created, each AZ will present a separate DNS address based on the EFS volume ID and the AZ short name as per below:

 

 

In order for an EC2 instance to mount an EFS partition it must first have the necessary NFSv4 client packages installed. There is a step by step walkthrough on the AWS site that details how to set up your environment so we won’t go through every step again here. However, the basic point is that all EC2 instances in a particular AZ are recommended to mount the DNS name of the EFS target in the same AZ. This is primarily for performance reasons but it is important to note if you have an autoscaling group that spans multiple AZs. You will need to query instance metadata to find out the AZ short name and cater for this in your build and configuration automation.

It is highly probably that AWS bake redundancy into the DNS target but they don’t publish the details on how this is achieved. We explore more about failover in an upcoming post.

Performance

Once we had everything set up and had performed some basic testing, we wanted to get an idea about the relative performance of EFS when compared to other AWS storage offerings as well as GlusterFS. Many thanks to Cloudten’s Shawn Zhang for configuring and running these tests.

All our testing was performed using AWS Linux EC2 clients using AMI amzn-ami-hvm-2015.09.01.x64_64-gp2 . If you want to make use of the Enhanced Networking feature of certain EC2 instance types then it is essential that you choose an HVM AMI.

We performed testing against the following types of storage:

  1. An 8GB magnetic EBS volume that was attached during instance creation.
  2. An 100GB magnetic EBS volume that was provisioned and attached outside of the instance build.
  3. An 100GB SSD EBS volume. Remember SSDs under 100GB are capable of occasionally  ‘bursting’ to 3000 PIOPS.
  4. A 100GB PIOPS enabled EBS volume with 3000 PIOPS.
  5. An EFS volume defined across two Availability Zones.
  6. An S3 bucket. Yes this is rather silly but we wanted to demonstrate why it’s not a filesystem.

But aren’t the first two the same thing you ask ? Yes, they should be, but we wanted to test out a theory that when you provision an EBS volume during EC2 instantiation that it comes from a SAN physically close to the EC2 hypervisor and will therefore have slightly better throughput. Hard to prove we know, but let’s give it a shot.

We also chose four different EC2 instance types:

EC2 Type Num CPU CPU Type RAM E/N Cost (USD p/h)
t2.micro 1 burstable Intel Xeon up to 3.3GHz 1 N 0.013
m3.medium 1 2.6 GHz Intel Xeon E5-2670 v2 3.75 N 0.067
m4.large 2 2.4 GHz Intel Xeon® E5-2676 v3 8 Y 0.12
c4.xlarge 4 2.9 Ghz Intel Xeon E5-2666 v3 7.5 Y 0.209

We didn’t pre-warm the EBS volumes and all instances were spun up specifically for the tests.

The first set of tests involved using the UNIX dd command to read and write multiple 1GB files between the EC2 instance and the target volume. For the S3 test we used the AWS CLI with a standard ‘cp’ command. Each operation was performed 5 times to create a single run. Each run was repeated 5 times with a sleep in between runs and all files cleared in between runs. The average results were recorded.

The following shows the results (in MBps) for the two smaller instance types. Orange shows the read speed and blue the write.

We see a few interesting things from these results:

  1. S3 read/write speed for large files isn’t great. This is as expected ( Yes, there are tools to make it better)
  2. The theory about instance created EBS volumes having lower latency appears to be a myth
  3. It is highly possible that our t2.micro was ‘bursting‘ during the test, but it did seem to outperform the m3.medium. Whilst this performance can’t be expected to be maintained consistently, it does show that t2 instances are capable of punching above their weight.

Let’s see what happens when we use EC2 instances with Enhanced Networking enabled. Depending on the actual type and instance size, these may also come with dedicated EBS bandwidth. If you do have high throughput requirements, then you would ideally be looking to use this feature which is only available on certain instance types. (C3,C4,D2,I2,M4,R3).

As you can clearly see Enhanced Networking makes a (very) big difference. We have included the results of the smaller instance tests above to highlight the scale of the improvement.

So what does this say about EFS ? It’s not as fast as a dedicated SSD or PIOPS enabled EBS volume but it isn’t really meant to be. The fact that its performance is generally comparable to a magnetic EBS volume is quite impressive for an NFS mounted filesystem. When you take into consideration that the data is being replicated and synchronised across multiple AZs and able to support multiple read/write operations across different clients, it is actually very good.

Perhaps a fairer test is to put EFS against a comparable shared filesystem such as GlusterFS. As we have already established the dominance of the c4.xlarge in our previous tests we will use that as the benchmark. Unfortunately we don’t know what AWS is using on the back end of EFS, but we do presume that they are using some form of compute nodes to present the network share similar to what Gluster does.

We decided to use c4.xlarge instances for 2 x Gluster nodes and attached 100GB 3000 PIOPS EBS volumes to each for the network share. This is a reasonably strong setup. The same c4.xlarge client node was used for both EFS and Gluster testing. The Gluster nodes were placed in different AZs with appropriate security groups.

Initially, the client node mounted both the EFS DNS target and Gluster node in the same AZ as the client as per recommendations. We repeated the previous test of transferring 1GB files to and from the EFS and Gluster targets one at a time. It is worth mentioning that the GlusterFS client, only uses the target IP address for initial connection and then internally figures out which node to send it to. However you would like to think it prefers the fastest route. We didn’t want to force traffic down one path by shutting down the alternate instance as this will stop the data replication which could also skew the results. The below shows the results of the 1GB files test that we performed previously

 

This shows results in favour of EFS. However, what if your potential use case for EFS is for much smaller files ? We used the same setup to run another test with 1000 x 2k files and tried to push them all through at speed. Arguably, these file sizes are probably too small and better suited to some kind of caching filesystem.

 

As you can see from the above, the results are quite different and definitely seem to favour GlusterFS. It is highly likely that your particular use case for a shared filesystem may involve a combination of file sizes however.

Price is where EFS really becomes attractive though. If we perform a cost comparison on the tests we just conducted, EFS comes out a big winner. As we mentioned earlier, EFS, like S3, does not require you to specify the size of your share in advance. It will dynamically grow as you put more files in there. If we are using EBS volumes then we basically need to provision for the future. If we know we immediately have a requirement for 10GB but that this will grown by 2GB a week then it makes sense to provision considerably more than the initial 10GB. In our example we used a 100GB EBS volume with dedicated 3000 PIOPS. These are relatively quite expensive so we will also include an example using standard SSD EBS volumes. We also need to account for the  2 x c4.xlarge EC2 instances that are needed to present them. We can bring this cost down quite a bit by using reserved instances, but the difference is still significant. If we put it into numbers, it looks like this:

Share Type Monthly Storage (10GB) Monthly Storage (100GB) Monthly Storage (1TB) EC2  (1yr-100%-reserved)
EFS $3 $30 $300 $0
GlusterFS (SSD) $2 $20 $200 $191.84
GlusterFS (PIOPS) $415* $415 $640 $191.84

All prices are in USD in the US West Oregon region where EFS is currently in Preview. For the PIOPS test if we want to hit our target of 3000PIOPS we must provision a minimum disk size of 100GB. The above pricing also factors in two EBS volumes and 2 EC2 instances for GlusterFS. If we put the above number into a graph we get the following.

In summary, EFS definitely seems to be a viable, cost effective and very comparable (from a performance perspective) shared filesystem. We are also performing failover and redundancy tests and will present our findings in a separate blog.

Thanks for reading. I know this has been a long one and thanks again to Shawn Zhang for creating and running the tests.

 

Update 1

We have had a lot of positive feedback about the post and a number of people have had similar experiences with smaller file sizes not performing particularly well with EFS.

As such we decided to run some more tests by tuning the NFS client settings on EFS as well as using an NFS  client rather than the recommended GlusterFS with GlusterFS.

As you can see, the native NFS client when used with GlusterFS does significantly decrease the throughput compared with the GlusterFS client. Tuning the client on the EFS side does not seem to improve it much


Cloudten Industries © is an Australian cloud practice and a recognised consulting partner of AWS. We specialise in the design, delivery and support of cloud based solutions. Don’t hesitate to contact us if you have any queries about this post or any cloud related topic.