Stanford University researchers have released a report assessing the transparency of artificial intelligence (AI) foundation models developed by companies such as OpenAI and Google. The report calls on these companies to disclose more information about their models, including details about the data and human labor used in their training processes.
According to the researchers, transparency in AI foundation models has declined in recent years, even as the capabilities of these models have grown significantly. Percy Liang, a Stanford professor involved in the Foundation Model Transparency Index, expressed concern about this trend, highlighting that reduced transparency can lead to negative consequences, as seen in other areas like social media.
Foundation models are AI systems trained on massive datasets, enabling them to perform a wide range of tasks, from writing to coding. Companies developing these foundation models are driving the rapid advancement of generative AI, attracting businesses of all sizes. As these models play an increasingly vital role in decision-making and automation, understanding their limitations and biases becomes crucial.
The Foundation Model Transparency Index evaluated ten popular models based on 100 transparency indicators, including training data and computational resources used. All models received relatively low scores, with even the most transparent model, Meta’s Llama 2, earning a score of 53 out of 100. Amazon’s Titan model had the lowest score, at 11 out of 100, while OpenAI’s GPT-4 received a score of 47 out of 100.
The authors of the index hope that the report will motivate companies to enhance transparency regarding their foundation models. Additionally, they believe it can serve as a starting point for governments and regulators as they grapple with how to oversee and regulate this rapidly evolving field.
The index is a collaborative effort from the Stanford Institute for Human-Centered Artificial Intelligence’s Center for Research on Foundation Models.