The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon; Lucile Saulnier; Thomas Wang; Christopher Akiki; Albert Villanova del Moral; Teven Le Scao; Leandro von Werra; Chenghao Mou; Eduardo González Ponferrada; Huu Nguyen; Jörg Frohberg; Mario Šaško; Quentin Lhoest; Angelina McMillan-Major; Gérard Dupont; Stella Biderman; Anna Rogers; Loubna Ben allal; Francesco De Toni; Giada Pistilli; Olivier Nguyen; Somaieh Nikpoor; Maraim Masoud; Pierre Colombo; Javier de la Rosa; Paulo Villegas; Tristan Thrush; Shayne Longpre; Sebastian Nagel; Leon Weber; Manuel Romero Muñoz; Jian Zhu; Daniel van Strien; Zaid Alyafeai; Khalid Almubarak; Vu Minh Chien; Itziar Gonzalez-Dios; Aitor Soroa; Kyle Lo; Manan Dey; Pedro Ortiz Suarez; Aaron Gokaslan; Shamik Bose; David Ifeoluwa Adelani; Long Phan; Hieu Tran; Ian Yu; Suhas Pai; Jenny Chim; Violette Lepercq; Suzana Ilić; Margaret Mitchell; Sasha Luccioni; Yacine Jernite

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Hugo Laurençon
, Lucile Saulnier
, Thomas Wang
, Christopher Akiki
, Albert Villanova del Moral
, Teven Le Scao
, Leandro von Werra
, Chenghao Mou
, Eduardo González Ponferrada
, Huu Nguyen
, Jörg Frohberg
, Mario Šaško
, Quentin Lhoest
, Angelina McMillan-Major
, Gérard Dupont
, Stella Biderman
, Anna Rogers
, Loubna Ben allal
, Francesco De Toni
, Giada Pistilli

Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Romero Muñoz, Jian Zhu, Daniel van Strien, Zaid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ifeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilić, Margaret Mitchell, Sasha Luccioni, Yacine Jernite

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

165 Scopus citations

Abstract

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM)(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Original language	English
Title of host publication	Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022
Editors	S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh
Publisher	Neural information processing systems foundation
ISBN (Electronic)	9781713871088
State	Published - 2022
Externally published	Yes
Event	36th Conference on Neural Information Processing Systems, NeurIPS 2022 - New Orleans, United States Duration: 28 Nov 2022 → 9 Dec 2022

Publication series

Name	Advances in Neural Information Processing Systems
Volume	35
ISSN (Print)	1049-5258

Conference

Conference	36th Conference on Neural Information Processing Systems, NeurIPS 2022
Country/Territory	United States
City	New Orleans
Period	28/11/22 → 9/12/22

Cite this

Laurençon, H., Saulnier, L., Wang, T., Akiki, C., del Moral, A. V., Le Scao, T., von Werra, L., Mou, C., Ponferrada, E. G., Nguyen, H., Frohberg, J., Šaško, M., Lhoest, Q., McMillan-Major, A., Dupont, G., Biderman, S., Rogers, A., Ben allal, L., De Toni, F., ... Jernite, Y. (2022). The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, & A. Oh (Eds.), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022 (Advances in Neural Information Processing Systems; Vol. 35). Neural information processing systems foundation.

Laurençon, Hugo ; Saulnier, Lucile ; Wang, Thomas et al. / The BigScience ROOTS Corpus : A 1.6TB Composite Multilingual Dataset. Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. editor / S. Koyejo ; S. Mohamed ; A. Agarwal ; D. Belgrave ; K. Cho ; A. Oh. Neural information processing systems foundation, 2022. (Advances in Neural Information Processing Systems).

@inproceedings{2143aa733e95441994b5880545f1cc0d,

title = "The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset",

abstract = "As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM)(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.",

author = "Hugo Lauren{\c c}on and Lucile Saulnier and Thomas Wang and Christopher Akiki and \{del Moral\}, \{Albert Villanova\} and \{Le Scao\}, Teven and \{von Werra\}, Leandro and Chenghao Mou and Ponferrada, \{Eduardo Gonz{\'a}lez\} and Huu Nguyen and J{\"o}rg Frohberg and Mario {\v S}a{\v s}ko and Quentin Lhoest and Angelina McMillan-Major and G{\'e}rard Dupont and Stella Biderman and Anna Rogers and \{Ben allal\}, Loubna and \{De Toni\}, Francesco and Giada Pistilli and Olivier Nguyen and Somaieh Nikpoor and Maraim Masoud and Pierre Colombo and \{de la Rosa\}, Javier and Paulo Villegas and Tristan Thrush and Shayne Longpre and Sebastian Nagel and Leon Weber and Mu{\~n}oz, \{Manuel Romero\} and Jian Zhu and \{van Strien\}, Daniel and Zaid Alyafeai and Khalid Almubarak and Chien, \{Vu Minh\} and Itziar Gonzalez-Dios and Aitor Soroa and Kyle Lo and Manan Dey and Suarez, \{Pedro Ortiz\} and Aaron Gokaslan and Shamik Bose and Adelani, \{David Ifeoluwa\} and Long Phan and Hieu Tran and Ian Yu and Suhas Pai and Jenny Chim and Violette Lepercq and Suzana Ili{\'c} and Margaret Mitchell and Sasha Luccioni and Yacine Jernite",

note = "Publisher Copyright: {\textcopyright} 2022 Neural information processing systems foundation. All rights reserved.; 36th Conference on Neural Information Processing Systems, NeurIPS 2022 ; Conference date: 28-11-2022 Through 09-12-2022",

year = "2022",

language = "English",

series = "Advances in Neural Information Processing Systems",

publisher = "Neural information processing systems foundation",

editor = "S. Koyejo and S. Mohamed and A. Agarwal and D. Belgrave and K. Cho and A. Oh",

booktitle = "Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022",

address = "United States",

}

Laurençon, H, Saulnier, L, Wang, T, Akiki, C, del Moral, AV, Le Scao, T, von Werra, L, Mou, C, Ponferrada, EG, Nguyen, H, Frohberg, J, Šaško, M, Lhoest, Q, McMillan-Major, A, Dupont, G, Biderman, S, Rogers, A, Ben allal, L, De Toni, F, Pistilli, G, Nguyen, O, Nikpoor, S, Masoud, M, Colombo, P, de la Rosa, J, Villegas, P, Thrush, T, Longpre, S, Nagel, S, Weber, L, Muñoz, MR, Zhu, J, van Strien, D, Alyafeai, Z, Almubarak, K, Chien, VM, Gonzalez-Dios, I, Soroa, A, Lo, K, Dey, M, Suarez, PO, Gokaslan, A, Bose, S, Adelani, DI, Phan, L, Tran, H, Yu, I, Pai, S, Chim, J, Lepercq, V, Ilić, S, Mitchell, M, Luccioni, S & Jernite, Y 2022, The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. in S Koyejo, S Mohamed, A Agarwal, D Belgrave, K Cho & A Oh (eds), Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Advances in Neural Information Processing Systems, vol. 35, Neural information processing systems foundation, 36th Conference on Neural Information Processing Systems, NeurIPS 2022, New Orleans, United States, 28/11/22.

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. / Laurençon, Hugo; Saulnier, Lucile; Wang, Thomas et al.
Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. ed. / S. Koyejo; S. Mohamed; A. Agarwal; D. Belgrave; K. Cho; A. Oh. Neural information processing systems foundation, 2022. (Advances in Neural Information Processing Systems; Vol. 35).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - The BigScience ROOTS Corpus

T2 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

AU - Laurençon, Hugo

AU - Saulnier, Lucile

AU - Wang, Thomas

AU - Akiki, Christopher

AU - del Moral, Albert Villanova

AU - Le Scao, Teven

AU - von Werra, Leandro

AU - Mou, Chenghao

AU - Ponferrada, Eduardo González

AU - Nguyen, Huu

AU - Frohberg, Jörg

AU - Šaško, Mario

AU - Lhoest, Quentin

AU - McMillan-Major, Angelina

AU - Dupont, Gérard

AU - Biderman, Stella

AU - Rogers, Anna

AU - Ben allal, Loubna

AU - De Toni, Francesco

AU - Pistilli, Giada

AU - Nguyen, Olivier

AU - Nikpoor, Somaieh

AU - Masoud, Maraim

AU - Colombo, Pierre

AU - de la Rosa, Javier

AU - Villegas, Paulo

AU - Thrush, Tristan

AU - Longpre, Shayne

AU - Nagel, Sebastian

AU - Weber, Leon

AU - Muñoz, Manuel Romero

AU - Zhu, Jian

AU - van Strien, Daniel

AU - Alyafeai, Zaid

AU - Almubarak, Khalid

AU - Chien, Vu Minh

AU - Gonzalez-Dios, Itziar

AU - Soroa, Aitor

AU - Lo, Kyle

AU - Dey, Manan

AU - Suarez, Pedro Ortiz

AU - Gokaslan, Aaron

AU - Bose, Shamik

AU - Adelani, David Ifeoluwa

AU - Phan, Long

AU - Tran, Hieu

AU - Yu, Ian

AU - Pai, Suhas

AU - Chim, Jenny

AU - Lepercq, Violette

AU - Ilić, Suzana

AU - Mitchell, Margaret

AU - Luccioni, Sasha

AU - Jernite, Yacine

PY - 2022

Y1 - 2022

N2 - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM)(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

AB - As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM)(BigScience Workshop, 2022) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

UR - https://www.scopus.com/pages/publications/85143055951

M3 - Conference contribution

AN - SCOPUS:85143055951

T3 - Advances in Neural Information Processing Systems

BT - Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022

A2 - Koyejo, S.

A2 - Mohamed, S.

A2 - Agarwal, A.

A2 - Belgrave, D.

A2 - Cho, K.

A2 - Oh, A.

PB - Neural information processing systems foundation

Y2 - 28 November 2022 through 9 December 2022

ER -

Laurençon H, Saulnier L, Wang T, Akiki C, del Moral AV, Le Scao T et al. The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset. In Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, editors, Advances in Neural Information Processing Systems 35 - 36th Conference on Neural Information Processing Systems, NeurIPS 2022. Neural information processing systems foundation. 2022. (Advances in Neural Information Processing Systems).

The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

Abstract

Publication series

Conference

Other files and links

Fingerprint

Cite this