Language Identification with Additive Attention BiLSTM and LaBSE Embeddings

Language Identification (LangID) plays a critical role in multilingual NLP pipelines, particularly for large-scale web crawling, dataset filtering, and pre-processing in tasks such as machine translation and cross-lingual retrieval. With the growing volume of web-scale multilingual content, robust and scalable LangID systems are essential for curating clean and language-specific data. In this paper, we propose a hybrid neural architecture that combines sentence-level semantic representations from LaBSE embeddings with token-level sequential features modeled by a BiLSTM with additive attention. This design captures both global context and local subword cues, enabling accurate language classification even in short or noisy texts. Evaluated on the WiLI-2018 dataset spanning 235 languages, our model achieves 88.37% accuracy—outperforming strong baselines like FastText and XLM-RoBERTa. The proposed approach offers a compelling balance between performance, scalability, and deployability for real-world multilingual data pipelines.

Note: this paper was accepted (CCIS/Springer short paper) at FDSE 2025, but was withdrawn and not submitted for final publication.