The Trade Desk achieves hyperscale for real-time bidding
About The Trade Desk
The Trade Desk is an advertising technology company that provides a self-service omni-channel platform through which media buyers can purchase digital advertising. The company was founded in 2009 and has outpaced growth in worldwide programmatic advertising, as of 2023 currently $1.73B revenue with a market capitalization of $36B.
Manage and leverage the firehose of 3rd party data sources for real-time bidding
The Trade Desk provides a real-time exchange between advertisers and publishers, and gives advertisers the opportunity to purchase ad placements as a targeted user loads a webpage. The platform combines real time situational data with anonymized data about the user audience and their interests to help advertisers get the most value from their decision and money spent.
As global internet traffic and advertising opportunities increase in volume, the Trade Desk needed a solution that could allow the company to run expanded numbers of targeted campaigns at petabyte-scale, overcome multiple challenges with Cassandra, and capitalize on opportunities that had originally been missed due to the time it took for user profile data to thaw.
High performance, real-time ad targeting
Analyze real-time and stored data to select which ads to show while preserving website performance and keeping CPU costs under control, at 12 million times a second.
High volume of data
Condense, read, store, and analyze data points from 10 billion records - each potentially having tens of thousands of data points or segments.
Storage optimization
Needed to create an offline “cold store” to store unused data and optimize for “thawing” for real-time systems.
Hyperscale, real-time bidding leveraging both hot cache and cold store
By moving off of Cassandra and onto Aerospike, the Trade Desk was able to reduce server instance footprint, gain flexibility for the existing and future use cases, and improve its ability to target customers.
Edge hot cache for real-time bidding
Aerospike receives 12 million queries per second (800 billion queries per day).
System of record on AWS
Manages peak loads of 20 million writes per second in its “cold storage” of user profiles.
Focus on business opportunities, versus data absorption
Increased allocation of CPU utilization on bidding processes allows The Trade Desk to take advantage of more opportunities in the market.
Record-based model
All data put into one record - which is less CPU-intensive.
Flexible data organization
Use one key to represent many different dimensions and get back only the data needed for a given use case.
Expanded bid opportunities, more efficient infrastructure, and reduced footprint
Using Aerospike as a hot cache with a cold store gives The Trade Desk the ability to capture more business, deliver more value to its ecosystem of advertisers, and keeps costs under control.
Efficient infrastructure with reduced footprint
The Trade Desk reduced server footprint from 500 to 60 machines while also reducing in-data center footprint.
Improved flexibility to address current and future use cases
Aerospike’s performance and technical capabilities opened up new flexibility for the current use case as well as other, desired use cases.
Rapid access to cold store
With Aerospike, the Trade Desk can thaw data in 8 milliseconds for real-time bidding.
Improved site performance
Data delivery impact on bidding site performance eliminated.
Opportunity for advanced analytics
Aerospike Connect for Spark combined with a cold storage cluster allows for advanced analytics.
Additional resources
For a deeper understanding and more insights, explore these additional resources.
- Video
The Trade Desk: From Cloud to On-Prem – Capacity Planning and CPU Analysis
The Trade Desk, the world's largest independent programmatic advertising DSP, migrated one of its largest Aerospike clusters from cloud to on-prem due to new business requirements, a tailor-built site, and risk and capacity challenges, with this session covering the findings and methods used to ensure confidence in the move.
- Video
The Trade Desk: Integrated Hyper-Scale Hot and Cold Store
The Trade Desk's Cold Storage uses Aerospike as a global hot cache and offline store, enabling petabyte-scale targeted campaigns and solving challenges previously faced with Cassandra.
- Video
The Trade Desk: Large Scale Cluster Management with Ansible
Aerospike automatically replicates data during node changes, but with large nodes, re-balancing can be slow; The Trade Desk automated this with Ansible on AWX, ensuring high availability.
Get started with Aerospike
For high-performance, scalable data management and ultra-low latency, ideal for handling massive datasets and real-time applications.
