If You Build Your Own NER Scorer, Non-replicable Results Will Come

Constantine Lignos; Marjan Kamyab

Back

If You Build Your Own NER Scorer, Non-replicable Results Will Come

Conference proceeding

Open access

If You Build Your Own NER Scorer, Non-replicable Results Will Come

Constantine Lignos and Marjan Kamyab

Proceedings of the First Workshop on Insights from Negative Results in NLP, pp.94-99

2020 Workshop on Insights from Negative Results in NLP, Association for Computational Linguistics, 1 (online/virtual, 11/16/2020 - 11/20/2020)

11/2020

Abstract

Named Entity Recognition

Computational Linguistics

Natural Language Programming

We attempt to replicate a named entity recognition (NER) model implemented in a popular toolkit and discover that a critical barrier to doing so is the inconsistent evaluation of improper label sequences. We define these sequences and examine how two scorers differ in their handling of them, finding that one approach produces F1 scores approximately 0.5 points higher on the CoNLL 2003 English development and test sets. We propose best practices to increase the replicability of NER evaluations by increasing transparency regarding the handling of improper label sequences.

Files and links (1)

url

If You Build Your Own NER Scorer, Non-replicable Results Will ComeView

Published (Version of record)"If You Build Your Own NER Scorer, Non-replicable Results Will Come", paper from Association for Computational Linguistics conference Open

Metrics

10 Record Views

Details

Title: If You Build Your Own NER Scorer, Non-replicable Results Will Come
Creators: Constantine Lignos (Author) - Brandeis University, Michtom School of Computer Science
Marjan Kamyab (Author)
Publication Details: Proceedings of the First Workshop on Insights from Negative Results in NLP, pp.94-99
Conference: 2020 Workshop on Insights from Negative Results in NLP, Association for Computational Linguistics, 1 (online/virtual, 11/16/2020 - 11/20/2020)
Publisher: Association for Computational Linguistics
Identifiers: 9924117867601921
Academic Unit: Benjamin and Mae Volen National Center for Complex Systems; Michtom School of Computer Science
Language: English
Resource Type: Conference proceeding

If You Build Your Own NER Scorer, Non-replicable Results Will Come

Abstract

Files and links (1)

Metrics

Details

Brandeis University Social media