Every day AppNexus data pipeline processes nearly 10 terabytes of data generated by over 17 billion ad requests (300k/second @ peak), and these numbers are growing rapidly. We use this data to create aggregated analytics reports, make ad campaign budget updates and drive optimization engines. This scale is the source of a number of engineering challenges both in terms of the amounts of data and the processing involved.

The AppNexus Data Team tackles these problems. We will be going over some of the key technologies we are using to tackle these challenges. We will be discussing how we introduced Hadoop into our ecosystem while processing this volume and required to be up 24/7.

Speakers:

Ersin Yilmaz, Engineering Director
Sateesh Lakkarsu, Sr. Software Engineer