We determine that the importance of the pre-training dataset size was largely overestimated, as we are able to repeatedly show that language models can be pre-trained with corpora of a modest size.

We consider several neural language models, some of which trained or fine-tuned on a new corpus of raw Old and Middle French texts, and use their internal representations of words as inputs to train taggers and parsers on the SRCMF treebank.

We present the $\text{FreEM}_{\text{max}}$ corpus of Early Modern French and D’AlemBERT, a RoBERTa-based language model trained on $\text{FreEM}_{\text{max}}$.

we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR.

We audit 5 multilingual corpora, finding that lower-resource corpora have systematic issues.

We opt for a data-driven approach by developing a new corpus with fine-grained entity annotation, covering three centuries of literature corresponding to the early modern period, We then fine-tune existing state-of-the-art architectures obtaining results that are on par with those of the current state-of-the-art NER systems for Contemporary English.

We present annotated corpora and NLP models for some downstream tasks in Early Modern French.

We developed our online catalogue as a supporting tool for gathering metadata through organized public hackathons.

We propose a new pipeline that is faster, modular, parameterizable, and well documented. We use it to create a corpus similar to OSCAR but larger and based on recent data.

In this article we present the approaches developed by the Sorbonne-INRIA for NER (SinNer) team for the CLEF-HIPE 2020 challenge on Named Entity Processing on old newspapers.

Nous présentons une pipeline pour le traitement et l’enrichissement de documents basée sur les dernières méthodes d’apprentissage neuronal.

We explore the impact of the training corpus on contextualized word embeddings in five mid-resource languages.

We propose a new pipeline to filter, clean and classify Common Crawl by language, we publish the final corpus under the name OSCAR.

A talk about automatic enrichment of dictionaries.