Thorn Technologies helps Experient leverage Amazon Web Services to collect and process massive amounts of event attendee location data faster.
The company provides these location insights to:
- Event planners to help them better understand how attendees travel throughout the event site
- Sponsors who purchase booth space at the event to provide a thorough understanding of how many attendees walked by, at what times, and how long they dwelled in certain areas.
Experient was preparing for one of the largest trade shows in the world, where they expected to collect 15 times more data than any previous event the company helped manage.
Thus, they needed to build a robust solution that could capture and process billions of data points quickly, which would help them provide their clients with more powerful reporting capabilities.
Experient engaged Thorn Technologies to help build this data capture and processing solution, and AWS played an integral role.
The first step was to identify how the current backend architecture could be improved.
We identified the data flow of the existing system as follows:
- Beacon location data was collected by mobile apps that ran on devices distributed throughout the event floor
- This data was then fed into an on-premise server with a MongoDB database
- A service aggregated the MongoDB data in batches, which was then fed into an on-premise SQL database
- The SQL database powered reports that were accessible via a web portal
The primary issue was that the data couldn’t be aggregated in MongoDB in an efficient manner. This was because the aggregation process required more computing resources than what Experient had readily available in their on-premise data center.
Cloud computing was an ideal solution to solve this problem, and the next step was to determine how to set up the new cloud architecture.
Instead of using the on-premise MongoDB solution, we implemented Amazon Kinesis Firehose to deliver the data directly into AWS and store it in Amazon Redshift for aggregation and analysis.
With Amazon Redshift, we could create a scalable database cluster that is geared toward performant data warehouse solutions involving a lot of data. The number of nodes in the cluster can be scaled up based on the expected data load of the event with simple configuration options on the AWS console. The cluster can also be scaled down when that power was no longer needed. Thus, the constraints of the on-premise data center capacity would no longer apply.
After Redshift was implemented, we created a custom ETL process to aggregate the raw data and then export that data to another independent Redshift cluster, which was used to serve data to the analysis web portal. Having multiple aggregation and analysis clusters allowed for scaling of each of the clusters independently based on those different needs.
Replacing the on-premise analysis SQL server with Redshift resulted in better query performance and a much faster reporting portal.
Once that primary problem was solved, the system was then optimized.
Experient’s existing on-premise solution compressed the beacon data before delivering it to their API in order to save bandwidth (a scarce resource at major events). Instead of having Experient rewrite that optimized process, we added an API Gateway layer that used a Lambda to process that compressed data and send it on to Firehose.
Thus, the mobile apps that were used to collect the beacon data just needed to update the endpoint to the new API Gateway endpoint, without having to worry about including AWS SDKs or transforming the data to fit what Firehose expects.
The new cloud architecture significantly increased the speed of data capture and processing for Experient’s software.
The system can now process at least 240,000 captures per second (approximately 3 times faster than prior solutions) in real-time on a 6-node cluster that costs only $1.50/hr.
Furthermore, the solution has the ability to scale on-demand to support multiple billions of data captures for events of any size.
The company successfully launched the improved system for the large trade show and it captured 963 million data points, which were aggregated down to 125 million reads.
We scaled the servers to accommodate our predictions for the show. Data scans took less than 3 minutes to go from end-to-end (mobile reader to analysis portal). More than 50,000 beacon scans were processed every minute in real-time. Real-time heat maps were generated with this data in the analysis portal that wowed the organizers and exhibitors. It was a success, executed with the minimum of on-site issues.
Experient continues to move more of their IT infrastructure to AWS with the help of Thorn Technologies.