Machine Learning (ML) delivers success across domains like autonomous driving, cancer detection, or natural language processing. However, all established methods need copious amounts of labeled training data to learn. Labeling is a costly, error-prone, and labor-intensive process generally performed manually. We need to address the human resources behind ML - the labeling workers. Hereby an investigation of structural aspects of such labeling tasks, i.e., the sequence of presentation, is adamant. Structural aspects should influence perceived variety and thus, following self-determination theory, have effects on psychological and performance outcomes. In two previous literature-based studies we provided descriptive and normative knowledge about the state-of-the-art, research streams, and design principles in interactive labeling research. Currently we investigate above mentioned structural considerations of labeling and its effects. Further projects in this research area investigated labeling of chemical structural formulas.

Labeling interface with an advertisement image and four emotional cues that have to be labeled non-mutually exclusive. Note how the labeling is prevented from progressing until all four cues have been assigned (yes or no).

Help functionality of the labeling interface with a definition, a positive and a negative example for each emotional cue, accessible at any time during the labeling interaction.

Publications

• Power to the Oracle? Design Principles for Interactive Labeling Systems in Machine Learning Nadj, M.; Knaeble, M.; Li, M. X.; Maedche, A.2020. Künstliche Intelligenz, 1–12. doi:10.1007/s13218-020-00634-1
• Oracle or Teacher? A Systematic Overview of Research on Interactive Labeling for Machine Learning Knaeble, M.; Nadj, M.; Maedche, A.2020. 15. Internationale Tagung Wirtschaftsinformatik (WI 2020), Potsdam, 9 - 11 März 2020, 2–16, GITO Verlag. doi:10.30844/wi_2020_a1-knaeble