Last update 29/04/2024
In this article, we share the steps that we took to migrate our storage from Elastic File System to S3 on AWS.
Figure 1. AWS EFS Full architecture diagram
- Complexity in the architecture that’s hindering progress and developer experience
- Performance decline after website generation
- Performance issues with AWS EFS
- Costs increasing month after month
- Outdated content being served to the user
Figure 2. S3 bucket policy for Cloudflar
The bucket has also CORS policy set to allow GET, POST and PUT requests. The policy configuration and CORS is like this:
Figure 3. S3 bucket CORS policy
On the Cloudflare side, we configured a CNAME (Canonical Name) record for each domain, directing requests to the exposed host URL of the AWS S3 bucket host that follows this pattern BUCKET_NAMET.s3-website.eu-central-1.amazonaws.com via proxy. This move to AWS S3 reduced one step in the request sequence diagram by removing Nginx, as mentioned, and at the same time, it reduced the risk of a downtime for our online stores since Cloudflare and AWS S3 are taking care of serving the sites instead of our own infrastructure.
Our infrastructure makes sure to update the site after every generation but that will not cause the site to go down at any time. Check out the simplified sequence diagram.
Figure 4. Serve static page sequence diagram
The architecture diagram has changed as well, and the full diagram is shown below.
Figure 5. Updated full architecture diagram
This architectural update posed a challenge in ensuring uninterrupted site availability during the implementation and go-live phases. To address this, we adopted an approach of delivering incremental updates to the production environment to facilitate a smooth transition.
The migration involved several sequential steps, each of which was deployed to production upon completion:
- Preparation of the infrastructure, involving the automated creation of AWS S3 buckets with their corresponding policies for all countries.
- Implementation of the upload to AWS S3 in conjunction with the existing setup, preparing the site for the impending architecture switch.
- Readying the provisioning of Cloudflare CNAME creation through automation for every domain.
- Gradual rollout of the updated architecture, serving the site from AWS S3 one country at a time.
- Cleanup of obsolete legacy code for both the application and infrastructure.
Improvements
From this move, as mentioned, there were several benefits and we are going to list them below with some figures on how much we improved the performance and reduced the costs, amongst other things.
We got a 20% increase in cached requests by having 2 layers of caching, one cache in S3 bucket and one in Cloudflare. The data is for the month of July 2023.
Figure 6. Cached requests per storage
Table 1. Number of requests that we handled in July 2023
- The pipeline execution time was reduced by 40% since we built our docker images in a more optimised way and deployment provisioning takes less time since it spins up less resources.
- The page load time performance was improvement by 6% due to 2 layers of cache
- The storage costs were reduced by 85% across all our AWS accounts. The graph below shows the trajectory of the costs with AWS EFS that kept increasing and how that changed with the move to AWS S3 where the fluctuation is not that big, and it stays in the same range. It is too soon to see a clear stabilization. We see the spikes in November 2023 and January 2024 due to the amount of traffic that we get in these two months.
Figure 7. AWS storage cost trajectory
As we dive deeper into the architecture of Emma to shed more light on how our systems and services operate, we invite you to join us on this enlightening journey, gaining valuable insights into the intricate workings of Emma's architecture and the innovative solutions that power our innovative systems and services to improve sleep.
