Meta believes that making LLaMA 2 open source will give it a competitive advantage over OpenAI.
Introduction
Meta is releasing its first large language model, LLaMA 2, to the public, for free. This is a significant move for the company, which has been playing catch-up to rivals like OpenAI in the AI chatbot space.
LLaMA 2 is a suite of AI models that includes versions of the model in different sizes, as well as a version that can be built into a chatbot. The model is available for download from Meta’s launch partners Microsoft Azure, Amazon Web Services, and Hugging Face.
Meta hopes that releasing LLaMA 2 will help it catch up to OpenAI and other rivals in the AI chatbot space. The company also believes that making the model open source will benefit the entire AI community by giving people more options for how to use it.
“This is a really, really big moment for us,” said Ahmad Al-Dahle, a vice president at Meta who is leading the company’s generative AI work. “This benefits the entire AI community and gives people options to go with closed-source approaches or open-source approaches for whatever suits their particular application.”
The release of LLaMA 2 is a significant move by Meta and could have a major impact on the AI chatbot space. The model is still under development, but it has the potential to be a powerful tool for businesses and individuals.
What’s the catch?
While Meta has made LLaMA 2 available to the public for free, there are still some caveats to consider. The company is not releasing information about the data set that was used to train the model, so there is no way to know for sure whether it includes copyrighted works or personal data. Additionally, LLaMA 2 shares the same problems that plague all large language models: it can generate falsehoods and offensive language.
Here are some specific examples of the caveats that have been raised about LLaMA 2:
- Data privacy
Meta is not releasing information about the data set that was used to train the model. Thus, there is no way to know for sure whether it includes copyrighted works or personal data. This could raise concerns about privacy and security, especially if the model is used to generate text that is sensitive or confidential.
- Bias
Large language models are trained on massive datasets of text and code, which can reflect the biases that exist in the real world. This means that LLaMA 2 could generate text that is biased or discriminatory.
- Toxicity
Large language models can also generate text that is offensive or harmful. This is because they are trained on datasets that contain a wide variety of language, including both positive and negative examples. Meta admits it did not remove toxic data from the data set that was used to train LLaMA 2, because it believes that this data could help the model to detect hate speech more effectively. However, this could also lead to the model generating offensive language itself.
Behind the scenes
Getting LLaMA 2 ready to launch required a lot of work to make the model safer and less likely to spew toxic falsehoods than its predecessor.
Meta had plenty of past mistakes to learn from. Its language model for science, Galactica, was taken offline after only three days. The previous LlaMA model was also leaked online, sparking criticism from politicians who questioned whether Meta was taking proper account of the risks associated with AI language models, such as disinformation and harassment.
To mitigate the risk of repeating these mistakes, Meta applied a mix of different machine-learning techniques aimed at improving helpfulness and safety. This included:
- Data cleaning
The data used to train LLaMA 2 was carefully cleaned to remove toxic and harmful content.
- Safety-specific training
The model was trained on a dataset of text that was specifically designed to teach it to avoid generating toxic and harmful content.
- Human feedback
The model was also fine-tuned using human feedback, making sure that it was generating text that was helpful and safe.
Meta’s approach to training LLaMA 2 was more complex than usual for generative AI models, says Sasha Luccioni, a researcher at AI startup Hugging Face. This is because Meta was taking a number of steps to mitigate the risks associated with these models.
Take away
The launch of LLaMA 2 is a significant step forward for Meta in its efforts to develop safe and responsible AI language models. However, it is important to remember that these models are still under development, and there is always the risk that they could be used to generate harmful content. It is therefore important to use these models with caution and to be aware of the risks involved.
