Hardness sampling: exploring instance hardness in pool-based active learning

Abstract

Active Learning (AL) techniques enable the creation of efficient models with minimal annotation effort by deciding which portions of the available data are worth learning. Pool-based AL (PAL) is a specific scenario in which instances within a pool of unlabeled data must be selected, labeled by an oracle, and incorporated into a subset of the pool to be used as a training set. The goal of PAL is to build a growing subset that is increasingly more representative of the problem at hand. However, the proper strategy for an optimal query of such instances is still an open question. In this paper, we resort to Hardness Measures (HMs) to enrich the current repertoire of PAL strategies available to address this question. HMs are metrics that employ the Instance Hardness (IH) concept to identify instances with a higher probability of being misclassified and have been successfully applied in areas such as meta-learning and explainable AI. Likewise, this study adds to this collective effort by exploring the use of IH in the context of AL, examining HMs as informativeness criteria for PAL, which led to a new PAL strategy called Hardness Sampling (HardS). We tested HardS across multiple datasets and learners, demonstrating its competitive performance compared to classical strategies such as Uncertainty Sampling, Expected Error Reduction, and Density-weighted methods. The results also highlighted the success of neighborhood-based measures, especially the ratio of the intra-class and extra-class distances at an instance level. Additionally, some tree-based and likelihood-based measures also showed promising performance.