In this thesis, I propose three possible strategies, incre-mental relabeling, importance-weighted label prediction and active Bayesian Networks.

Use your “gold standard” data to measure the performance ofeach contributor so you know when to retrain workers. Whena contributor’s score falls below 70% accuracy, exclude hiswork and retrain.