This month we have Jeremy Stanley from Collective presenting “But does it really work? Using pervasive experimentation & visualization to hone machine learning on big data.”

Abstract:
How do you know if your machine learning system works? You’ve tested it on a held out set of data, but is that enough? What if you have 1,000s of models in production influencing billions of decisions in a chaotic, real time environment? What if you have to convince other internal departments critical to implementation success, your CEO, your customers?

At Collective, we have found that a combination of continuous ‘live’ experimentation, coupled with copious visualization have been key to our successes with machine learning in practice. This talk will cover our Audience Optimization system, which impacts billions of advertising impressions monthly. We continuously optimize thousands of campaigns over hundreds of millions of users, in multiple continents, against varying performance objectives.

Specifically, the talk will focus on:

The problems Audience Optimization addresses

How the system is used in production

The process and supporting architecture

The specific algorithms used

How we experiment with & visualize (almost) everything, always

How we prove our system works (to ourselves and our clients)

How we glean insights from visualizations to improve the system

The majority of the time will be spent on the final 3 points. This presentation should be of interest to any practitioners who are seeking to implement and continuously improve a large scale production machine learning system on big data.