Abstract
Shapley value-based data valuation methods, originating from cooperative game
theory, quantify the usefulness of each individual sample by considering its
contribution to all possible training subsets. Despite their extensive
applications, these methods encounter the challenge of value inflation - while
samples with negative Shapley values are detrimental, some with positive values
can also be harmful. This challenge prompts two fundamental questions: the
suitability of zero as a threshold for distinguishing detrimental from
beneficial samples and the determination of an appropriate threshold. To
address these questions, we focus on KNN-Shapley and propose Calibrated
KNN-Shapley (CKNN-Shapley), which calibrates zero as the threshold to
distinguish detrimental samples from beneficial ones by mitigating the negative
effects of small-sized training subsets. Through extensive experiments, we
demonstrate the effectiveness of CKNN-Shapley in alleviating data valuation
inflation, detecting detrimental samples, and assessing data quality. We also
extend our approach beyond conventional classification settings, applying it to
diverse and practical scenarios such as learning with mislabeled data, online
learning with stream data, and active learning for label annotation.