As outlined in our previous article discussing the architecture of Emma Online Store, we utilize AWS (Amazon Web Services) Elastic File System (EFS) as storage to deliver statically generated sites (HTML, CSS and JS files) for all countries. This is accomplished using Nginx as a proxy where requests first reach Cloudflare which forwards them to Nginx and Nginx reads the necessary files from the storage and serves them back to Cloudflare which serves it back to the user, as illustrated in the diagram below. The full article can be found here.
Challenges
Although the architecture that we had in place functioned effectively, we identified several issues that hindered our progress and limited our ability to expand the range of features. This complexity posed a potential challenge to the developer experience.
In addition to the impact on the developer experience, we observed a decline in performance during the hit of the first request after each generation. Despite utilising Cloudflare to cache our requests, we consistently cleared the cache following each website generation.
The performance dip was attributed to AWS EFS relying on a network file system beneath the surface, resulting in I/O operations when reading a file from a mounted volume in a running Kubernetes pod.
There was also the cost factor to consider since the implemented architecture was becoming more expensive every month since we kept all the generated versions of the site for all countries since the day when the architecture was implemented. There were no deletion policies in place and that resulted in increased storage costs.
Due to missing deletion policies, there were a lot of cases where we would serve outdated versions of a specific page to customers that had that page cached in their browser.
Summarising, we faced the following challenges:
Complexity in the architecture that’s hindering progress and developer experience
Performance decline after website generation
Performance issues with AWS EFS
Costs increasing month after month
Outdated content being served to the user
Solution
Considering the previously outlined issues and after conducting research, we opted to transition from AWS EFS to AWS S3. In this transition, our objective was to eliminate AWS EFS and Nginx as a proxy. Instead of transferring the statically generated files to AWS EFS during generation, we utilized the AWS SDK to push the files to AWS S3, where each country has its dedicated bucket.
The AWS S3 bucket is set up for static website hosting with object ownership, providing an exposed host URL. It is configured as private, with inline policy rules granting Cloudflare the necessary permissions to call the exposed host URL. The configuration of bucket policy allows the list of IPs from Cloudflare to access the bucket and the list of IPs can be found here.
The bucket has also CORS policy set to allow GET, POST and PUT requests. The policy configuration and CORS is like this:
On the Cloudflare side, we configured a CNAME (Canonical Name) record for each domain, directing requests to the exposed host URL of the AWS S3 bucket host that follows this pattern BUCKET_NAMET.s3-website.eu-central-1.amazonaws.com via proxy. This move to AWS S3 reduced one step in the request sequence diagram by removing Nginx, as mentioned, and at the same time, it reduced the risk of a downtime for our online stores since Cloudflare and AWS S3 are taking care of serving the sites instead of our own infrastructure.
Our infrastructure makes sure to update the site after every generation but that will not cause the site to go down at any time. Check out the simplified sequence diagram.
The architecture diagram has changed as well, and the full diagram is shown below.
This architectural update posed a challenge in ensuring uninterrupted site availability during the implementation and go-live phases. To address this, we adopted an approach of delivering incremental updates to the production environment to facilitate a smooth transition.
The migration involved several sequential steps, each of which was deployed to production upon completion:
Preparation of the infrastructure, involving the automated creation of AWS S3 buckets with their corresponding policies for all countries.
Implementation of the upload to AWS S3 in conjunction with the existing setup, preparing the site for the impending architecture switch.
Readying the provisioning of Cloudflare CNAME creation through automation for every domain.
Gradual rollout of the updated architecture, serving the site from AWS S3 one country at a time.
Cleanup of obsolete legacy code for both the application and infrastructure.
Improvements
From this move, as mentioned, there were several benefits and we are going to list them below with some figures on how much we improved the performance and reduced the costs, amongst other things.
- We got a 20% increase in cached requests by having 2 layers of caching, one cache in S3 bucket and one in Cloudflare. The data is for the month of July 2023.
- The pipeline execution time was reduced by 40% since we built our docker images in a more optimised way and deployment provisioning takes less time since it spins up less resources.
The page load time performance was improvement by 6% due to 2 layers of cache
The storage costs were reduced by 85% across all our AWS accounts. The graph below shows the trajectory of the costs with AWS EFS that kept increasing and how that changed with the move to AWS S3 where the fluctuation is not that big, and it stays in the same range. It is too soon to see a clear stabilization. We see the spikes in November 2023 and January 2024 due to the amount of traffic that we get in these two months.
As we dive deeper into the architecture of Emma to shed more light on how our systems and services operate, we invite you to join us on this enlightening journey, gaining valuable insights into the intricate workings of Emma's architecture and the innovative solutions that power our innovative systems and services to improve sleep.
Comments