From 2021 to 2022, over 1000 researchers from 250 institutions around the world came together for a workshop on large language models (LLMs) organized by the BigScience project. Its main achievement was BLOOM (“BigScience Large Open-science Open-access Multilingual Language Model”), a transformer-based model trained on 176 billion parameters.
While BLOOM’s underpinnings are similar to GPT-3, the language model underlying the now famous ChatGPT, as well as LaMDA, which powers Google’s Bard, there is a key difference.
The development of these commercial language models had been well underway at the time of the workshop. Therefore, the BigScience researchers chose a different approach to developing and training BLOOM, making this open-access model stand out from the rest.
How is BLOOM different from GPT-3 and LaMDA?
There are three main differences between BLOOM and the language models of OpenAI and Google: the open-source approach, the focus on ethical AI, and how it handles cultural diversity.
Let’s take a closer look at each of them.
1. BLOOM is open-source
Whereas GPT-3 and LaMDA are black boxes, providing only as much insight into their training data and algorithms as their developers deem necessary, BLOOM’s source code is readily available for anyone to inspect. Users can download the whole model free of charge and integrate it into their own projects, provided they have the computing power to actually run it.
Its open-access nature makes the model attractive for non-profit projects. Since relying on a commercial API like ChatGPT’s is not always financially feasible here, BLOOM represents a good alternative. This encourages a higher diversity of projects, with goals different from the many emerging AI-based business solutions.
2. BLOOM focuses on ethical use
The developers of various generative AI models have been criticized for using the creative works of others as training data without permission. There are also many ways in which LLMs can be used for less than noble endeavors, making them potentially dangerous tools.
The BLOOM team took both of these pitfalls into consideration when developing their language model. Instead of just scraping the internet for as much data as possible – and so reproducing its biases in the process – the researchers obtained information using pre-defined guidelines and made their sources available online.
They also created the Responsible AI License, which aims to regulate the usage of BLOOM to a certain degree. Whereas issues like intellectual property and distribution rights are still subject to debate for generative AI models, BLOOM comes with clearly defined usage rules.
3. BLOOM speaks 46 languages
From the beginning, diversity has been a major focus of the BigScience project, uniting researchers from around the globe and incorporating their influences into the final product. This is also reflected in the BLOOM language model, which can interact with users in 46 different languages, including 13 Indic and 20 African languages. It also knows 13 programming languages.
The number of supported languages is not what sets it apart from GPT-3 and LaMDA, however – it’s a better representation of cultural diversity. Researchers from all over the world carefully constructed high-quality data sets, thus providing BLOOM not just with multilingual input, but also with different points of view on the world.
Making the case for open-source AI
While BLOOM is a prime example of developing new technologies in the open, there are other examples of open-source models.
Meta disclosed the code of its OPT-175B, which derives its name from the 175 billion parameters it was trained on. Meanwhile, OpenAI open-sourced the neural network Whisper, which anyone with the necessary technical knowledge can use for transcribing spoken language.
This may bode well for a future in which freely accessible AI models coexist with their commercial counterparts.
Want to stay updated on the latest developments in machine learning and computer vision? Then keep an eye on our blog for more on these topics!