U.S. News & World Report

The Situation

Founded in 1933, U.S. News & World Report is a multi-platform publisher of news and information, which includes www.usnews.com and annual print and e-book versions of its authoritative rankings of Best Colleges, Best Graduate Schools and Best Hospitals. In 2012, U.S. News launched a conference division focusing on important national conversations and solutions in STEM Education and Healthcare of Tomorrow. U.S. News has earned a reputation as the leading provider of service news and information that improves the quality of life of its readers by focusing on health, personal finance, education, travel, cars, news and opinion. U.S. News & World Report’s signature franchises include its Consumer Advice and "Best" series of consumer guides that include rankings of colleges, graduate schools, high schools, hospitals, nursing homes, mutual funds, health plans, diets and more.

The Challenge

As a digital-centric media organization, U.S. News & World Report relies on its unique and compelling content to attract visitors to its website. It currently reaches more than 30 million decision makers every month, with 160 million page views representing a 20 percent year-over-year (YoY) increase.¹ U.S. News monetizes this content by selling advertisements and sponsorship opportunities to businesses seeking to reach its audience of educated professionals, business leaders and policy makers.

U.S. News’ intellectual property includes highly anticipated rankings in an increasing number of areas. For U.S. News, having this content freely available online makes it easier for its audience to find and helps improve its organic search rankings. However, it also makes it easier for unaffiliated third-parties to scrape and repurpose that same content on their sites using automated bots. This diverts visitors and can potentially impact advertising revenue. According to Matt Kupferman, Senior Director of Engineering, “U.S. News & World Report has actually seen independent parties taking our data, publishing it themselves and appearing higher in the search rankings.”

The Goals

U.S. News & World Report needed to meet several requirements to support its business objectives:

• Revenue protection. U.S. News wanted to detect and prevent third parties from scraping its website and repurposing its rankings data elsewhere.
• Reduced overhead with greater bot visibility. Without a solution in place, U.S. News had to manually sift through and analyze logs to identify and block bots that were scraping its site.
• Performance at scale. With 20% YoY growth, U.S. News wanted to ensure that the origin load did not compromise performance for legitimate users.
• Comprehensive bot management. U.S. News wanted granular visibility into bot traffic to better understand who was coming to its site and drive its bot management strategy.

Why Akamai?

Protecting and Enhancing Digital Revenue
For U.S. News, having the ability to protect its vast trove of rankings data from web scraping was the biggest business driver for deploying the Akamai Bot Manager solution. Kupferman says, “By ensuring we are the authoritative source of our data, we are funneling more of the organic search traffic into our website where we can monetize those views.” Protecting advertisement and sponsorship revenue from dilution by third-party sites directly supports its business and helps ensure that U.S. News remains financially viable in an all-digital world.

In addition, deploying Bot Manager enabled U.S. News to be in a position where health care data providers are willing to release additional data for inclusion in its health care content. Finding new ways to enhance the value of its content is critical to U.S. News’ strategy to continue growing its readership base and attracting an audience valuable to its advertising partners.

Greater Visibility into Bots
As with any online business, gaining greater visibility into the traffic coming to its website was a huge benefit for U.S. News. According to Kupferman, “gaining insight into who’s crawling your site is really interesting. Even in monitor-mode, you learn so much about where your traffic is coming from.” Having greater visibility into bot traffic —easily and in real-time through the Akamai Control Center —helped U.S. News better understand how the content it creates is being consumed by other third parties.

Scaling with the Business, in the Cloud
Prior to choosing Bot Manager, U.S. News had attempted to manually manage its bot traffic with their existing Web Application Firewall (WAF), and even briefly considered building its own in-house solution. When doing its due diligence on Akamai, it also investigated other bot management solutions that all had to be deployed on-premises inside its data center. However, the fact that Bot Manager runs on the same platform as its CDN was a huge advantage for U.S. News.

Kupferman explains, “We have a lot of non-unique requests, so we rely heavily on caching. Bot Manager was the only solution that operates at the CDN-level. Other solutions that we looked at required us to forgo caching. Now, we don’t have to.” With Bot Manager. U.S. News was able to reduce the load on its origin by 20%, reducing infrastructure costs and maintaining website performance for its readers. And, by better managing the IT impact of its bot traffic, U.S. News was able to better scale its web infrastructure to support the growth in its business more cost effectively.

A Turnkey Solution
The advantage of integration with the Akamai CDN was not limited to the reduction in origin load. From an IT perspective, not having to change the footprint of the existing infrastructure in any way was equally attractive. “Being a turnkey solution without having to re-architect everything was critical. With Bot Manager, we could flip a switch and have immediate visibility. For real integration at the edge, it was the only thing that fit the bill,” concludes Kupferman.