Machine Learning Seminar: More For Less: Adaptive Labeling Payments in Online Labor Markets

In many predictive tasks where human intelligence is needed to label training instances, online crowdsourcing markets have emerged as promising platforms for large-scale, cost-effective labeling. However, these platforms also introduce challenges that must be addressed for these opportunities to materialize. In particular, it has been shown that different trade-offs arise between payment offered to labelers and the quality of labeling at different times, possibly as a result of different market conditions and even the nature of the tasks themselves. Because the underlying mechanism giving rise to different trade-offs is not well understood, for any given labeling task and at any given time, it is unknown which labeling payments to offer in the market so as to produce accurate models cost-effectively. Effective and robust methods for dealing with these challenges are essential to enable a growing reliance on these promising and increasingly popular labor markets for large-scale labeling. In this talk I will first present a new data science problem, Adaptive Labeling Payment (ALP): how to learn and sequentially adapt the payment offered to crowd labelers before they undertake a labeling task, so as to produce a given (machine learning) predictive model performance cost-effectively. I will then present our approach to address the problem and a rich set of results that explore our approach's performance under a variety of market settings.