Holiday Readiness, Part Two — What you Should be Thinking About Three Months Out: Capacity Planning

This is the second post in a blog series about Akamai solutions that can help you manage the surge of traffic (both good and bad) that will be hitting the retail industry during the holiday season. Read part one and part three.

Welcome back to the Holiday Readiness blog series. We hope part one has kept you busy over the past month as you continue to improve your security posture. If you haven’t finished all of the security checklist items, don’t worry — there is still time before Black Friday and Cyber Monday.

For this blog, we will be talking about managing flash crowds, disaster recovery (DR), and communication strategies. When we have these conversations with our customers, there are many directions they can go in, primarily because each customer’s infrastructure and strategy are different. Some customers may have a single data center serving all of their content, some might use two, and others may use at least three scattered all over the world. No matter the type of setup, there are four main questions that you should be asking yourself as you prepare for the rush of holiday traffic:

1. Can my current infrastructure handle at least two to six times the amount of traffic (sustained) compared to a normal day?

2. If the Product or Marketing team promotes a major deal/discount, will my websites and applications be ready to handle the burst of short-term traffic?

3. Are my websites and applications set up for failover in the event there are issues?

4. Do I have sufficient alerts and monitoring in place to catch issues quickly?

As you read through this blog, each of these questions will be addressed so you can start having conversations with your teams as you prepare for the busiest time of the year.  

Stress/load testing

Year over year, there’s no question that more people are coming online and are continuously connected across thousands of devices. The internet in general is more accessible than ever as modern ISPs expand rural coverage via satellite connections at speeds that rival their terrestrial competitors. Coupled with the fact that the COVID-19 pandemic has forced millions to work from home over the past year and a half, online deals and flash sales are even more popular than before. For instance, in April 2020, Akamai observed a 30% increase in global internet traffic in just a few short weeks, which is the equivalent of an entire year’s worth of growth in internet traffic. If you haven’t started already, now is the time to begin setting up stress and/or load tests to identify any weak points with a solution such as Akamai CloudTest.

Information gathering

Start to think about the critical functions of your website or application that need to remain functional during the increased traffic we’re going to see in November. If you are reading this blog, there is a good chance you are already using a CDN like Akamai to reduce your origin server load by caching static objects and pages at the edge. But what about truly non-cacheable content? Things like personalization, certain database calls, and the checkout flow are all critical components that are extremely difficult, if not impossible, to cache due to the unique nature of the content being delivered to the end user. And since they are not cacheable, that means those requests need to make their way back to your infrastructure every single time.  

Now is also a good time to review any major site changes from the previous year. If there has been new functionality added to the website, does it need to be tested to see how well it will handle a sustained high server load? 

Review traffic throughout the year

Understanding your traffic levels over the past year will help determine what your baseline should be for testing this year. If you are utilizing Akamai mPulse for real user monitoring, you can look back at up to 13 months of your historical traffic, which will give you more than enough data to help with that determination.

Once you have a view of last year’s traffic, identify the weeks or months that sustained the highest traffic levels to develop an idea of where your baseline should be. Also, review any reports or graphs you may have from last year to determine whether there were issues at those previous load levels, so adjustments can be made.

Time to test

Given how uniquely customer websites and applications can be built, there is no definitive number of concurrent users that you should be working toward handling. Akamai has been supporting our retail customers throughout the holiday season for years, and we typically observe retail traffic increases anywhere between two to six times compared to baseline.

We’ve covered load testing and handling sustained traffic levels which, with the right setup, infrastructure can ultimately scale to support. What about flash crowds?

Managing flash crowds

Let’s start with an example to set the tone: At the last minute, the Marketing/Product teams decide they want to generate a lot of interest for a new product they are promoting with a special 50% off discount. They partner with a celebrity, who has millions of followers across all of their social media channels, to create a post and a video promoting the product. In addition to that, the Marketing team will be deploying millions of emails and text messages to their loyal customers at the same time. They have never run this type of deal before, and it will only last for one hour, so the team expects demand to be very high. At 1 PM, the team who manages the website receives an email from Marketing that this huge event is taking place the same day … an hour later at 2 PM.

The ending to that example can result in a few different outcomes, depending on how prepared they were to handle a short, volumetric spike in traffic to their infrastructure:

1. They have a solution in place to throttle incoming traffic in a controlled manner.

2. They were unprepared and the website experienced an outage.

3. Their infrastructure is already built to handle a large influx of traffic.

Below we will talk about the first two outcomes in a bit more detail. If your infrastructure is already built in a way to scale for flash crowds, then you’re already doing a great job! However, the incurred costs of spinning up and maintaining the additional infrastructure is something to think about.

Traffic throttling

In the “Stress/load testing” section above, we briefly touched on caching and how a CDN like Akamai can help reduce server load by storing cacheable content at the edge. But a critical workflow such as checkout can be quite difficult, if not impossible, to cache. Therefore, you may want to explore implementing a traffic throttling solution such as Visitor Prioritization, which will allow you to selectively control the amount of incoming traffic to your infrastructure.

By rerouting your end users to a customer-branded virtual “waiting room” that is hosted on Akamai NetStorage, you’ll have peace of mind knowing that your infrastructure will remain healthy while still allowing users to purchase their items in a controlled fashion. During the event, keep an eye on your infrastructure health to determine if you need to throttle up or down. Keep in mind, in addition to the Akamai Control Center user interface, you can also use our Cloudlets API or Visitor Prioritization CLI to adjust threshold levels programmatically.

Traffic control strategies

As you think about traffic throttling, a common question from customers is “How much traffic should I initially let in?” This can be a difficult question to answer unless you’re aware of all of the event details. If you’re unsure of the amount of traffic, then your best bet is to take the conservative approach and allow 0% to 25% of traffic in. Once the event begins and you have a better idea of traffic demands, you can increase the threshold incrementally while monitoring your server health.

Another strategy to think about using Visitor Prioritization outside of the flash crowd use case is in a disaster recovery scenario (read more about DR strategies below). If your infrastructure has already experienced issues and you’re currently experiencing an outage, you can repurpose Visitor Prioritization to ease traffic back onto your website as your servers recover. This still allows you to provide a branded experience to your end users while you incrementally allow traffic back onto the website.

Pro Tip: Some of our customers keep Visitor Prioritization enabled in an “always on” state, but allow 100% of the traffic through. This allows them to use the solution as a “panic button,” so to speak, to enable the virtual waiting room quickly.

Site failover and origin health

Unfortunately, there are occurrences where things can go wrong on the origin. A component can fail, and there is no backup database or throttling solution in place to reroute traffic and keep sales flowing. If that is the case, the next best thing you can do is serve an error or maintenance page from Akamai NetStorage. Even though the website may be down, users will still have a branded experience informing them that there are site issues while teams work to correct them.

Site Failover is a feature included with Ion, the solution that enables routing your website/application through the Akamai CDN. The image below shows a basic example of how to configure it in the event there are issues with your infrastructure. In this example, the Akamai edge server will react to either an origin connection timeout or when it receives an HTTP response status code matching 500 through 504. When this happens, users will be routed to Akamai NetStorage, and the file “page.html” within the maintenance directory will be displayed to the end user. Page.html is a basic HTML page that you have total creative control over. For more advanced implementations or questions, reach out to your Akamai account team.

If you already have Site Failover configured for your website, now is a good time to review and make any necessary changes to the content on your maintenance page. For example, some of our customers will add holiday flair to their maintenance pages to maintain a consistent theme or look if they update their website to use a holiday theme.

High availability

While the term “high availability” is normally attributed to availability across data centers, an Akamai feature that provides highly available routing is called SureRoute, which is part of Ion and Dynamic Site Accelerator. SureRoute is a feature that is designed to identify the fastest path across our network between an end user and your infrastructure. A key feature of SureRoute is that it is also continuously monitoring congestion and outages across the internet. When high congestion or an outage is identified, SureRoute will intelligently route traffic around those degraded areas when or if they occur. For example, if an underwater internet cable is suddenly cut, Akamai will automatically find the next best available route to your infrastructure. It may not be the fastest route at that time, but the user will still be able to browse the website or application.

Our edge servers do this by running “races” to a single, small, uncacheable file deployed on your infrastructure across multiple potential Akamai routes. (Typically, if you have a load balancer in front of your infrastructure, the file can be deployed there. Otherwise, that file should be placed on each origin server that you define in your Property Manager configuration.) Once the race is complete, the edge server measures which route is the fastest and will use that route for the next 30 minutes (this is configurable). Users who connect to the same edge server as others will take advantage of that same route since it was determined to be the fastest path from that location.

Monitoring and alerting

Monitoring your traffic and being notified when something unplanned happens are critical components that not only contribute to DR strategy, but also allow you to understand the overall health of your website or application over time. Within the Akamai Control Center, we have a number of alerts that you can configure to monitor the health of your website on client connections to the edge and also to help you monitor issues occurring on your origin. When an alert is triggered, you will be notified via email or SMS.

Reactive alerts

Below is an example of the various alerts that can be configured to monitor the health of your origin. This should by no means replace your existing Application Performance Monitoring tooling. However, implementing the alerts below to notify you of connection timeouts or a high rate of 5xx errors will help you confirm that there is indeed an issue so you can react quickly.