PAIR (People + AI Research) was first launched in 2017 with the belief that “AI can go much further and be more useful to all of us if we build systems with that in mind early in the process.” We remain focused on making AI more understandable, interpretable, fun and usable for more people around the world. It’s a mission that’s especially timely given the rise of generative AI and chatbots.
Today, PAIR is part of Google Research’s Responsible AI and Human-Centered Technologies team, and our work spans this larger research area: we publish educational materials, including the PAIR guide and Explorables (such as the recent Explorable, which looks at how and why models sometimes reliably make wrong predictions); and we develop software tools like the Learning Interpretability Tool that help people understand and debug ML behavior. This year our inspiration is to “change the way people think about what THEY can do with AI.” This vision is inspired by the rapid emergence of generative AI technologies, such as the large language models (LLM) that power chatbots like Bard, and new generative media models such as Google’s Imagen, Parti and MusicLM. In this blog post, we review PAIR’s recent work that is changing the way we interact with AI.
Generative AI research
Generative AI is generating a lot of excitement, and PAIR is involved in a range of related research, from using language models to build generative agents to exploring how artists have adopted generative image models such as Imagen and Parti the These latter “text-to-image” models allow a person to enter a text-based image description for the model (for example, “honeycomb house in the woods in a cartoon style”). In a forthcoming paper titled “The Prompt Artists” (Creativity and Cognition 2023), we found that users of generative imaging models seek not only to create beautiful images, but also to create unique, innovative styles. To achieve these styles, some will even seek out a unique vocabulary to help develop their visual style. For example, they can visit architecture blogs to learn which specific domain vocabulary they can adopt to help create distinctive images of buildings.
We’re also exploring solutions to the challenges faced by rapid developers who use generative AI to primarily program without using a programming language. As an example, we have developed new methods for extracting semantically meaningful structure from natural language prompts. We’ve implemented these constructs to instruct editors to provide functionality similar to that found in other programming environments, such as semantic highlighting, autosuggestion, and structured data views.
The rise of generative LLMs has also opened up new techniques for solving important long-term problems. Agile classifiers are one approach we use to leverage the semantic and syntactic strengths of LLMs to solve classification problems related to safer online conversation, such as blocking new types of toxic language as quickly as it can develop online. The big advance here is the ability to develop high-quality classifiers from very small data sets, smaller than 80 examples. This suggests a positive future for online discourse and its better moderation; instead of collecting millions of examples trying to create universal security classifiers for all use cases over months or years, more agile classifiers can be created by individuals or small organizations and tailored to their specific use cases, and iterated and adapted within a day (e.g. new types of intrusions blocking or to correct unanticipated bias in models). As an example of their utility, these methods recently won a SemEval competition to identify and explain sexism.
We have also developed novel state-of-the-art explanatory methods to explore the role of training data in modeling behavior and misbehavior. By combining training data imputation methods with agile classifiers, we also found that we could identify mislabeled training examples. This enables the reduction of noise in the training data, leading to significant improvements in model accuracy.
Overall, these methods are important in helping the scientific community improve generative models. They provide fast and efficient content control and dialog security techniques that help support creators whose content is the foundation for amazing results from generative models. Additionally, they provide straightforward tools to help debug model misbehavior, resulting in better generation.
Visualization and education
To reduce barriers to understanding ML-related work, we regularly design and publish highly visual, interactive online essays called AI Explorables that provide accessible and practical ways to learn about key ML concepts. For example, we recently published a new AI Explorables on the topics of model trust and unintended bias. In our latest Explorable, From Confidently Wrong Models to Humble Ensembles, we discuss the problem with model confidence; models can sometimes be a lot confident in their predictions… and yet completely wrong. Why does this happen and what can be done about it? Our Explorable walks through these issues with interactive examples and shows how we can build models that have more appropriate confidence in their predictions using a technique called ensemble, which works by averaging the results of multiple models. : Another study, “Looking for Specular Bias,” shows how spurious correlations can lead to unintended biases, and how techniques such as highlight maps can detect some biases in data sets, warning that the bias is hard to see. , when it is more. in a subtle and random training set.
 |
| PAIR designs and publishes AI Explorables, interactive essays on timely topics and new methods in ML research, such as From Confidently Incorrect Models to Humble Ensembles, which looks at how and why models make incorrect predictions with high confidence, and how is the “ensemble”? The results of many models can help avoid this. |
Transparency and the data card playbook
As we continue to advance our goal of helping people understand ML, we promote transparent documentation. In the past, PAIR and Google Cloud have developed model cards. Most recently, we presented our work on Data Cards at ACM FAccT’22 and the Open Source Data Cards Handbook, a collaborative effort with the Technology, AI, Society, and Culture team (TASC). The Data Cards Playbook is a toolkit of participatory activities and frameworks that help teams and organizations overcome barriers to transparency efforts. It was created using an iterative, multidisciplinary approach based on the expertise of more than 20 teams at Google and has four modules: Ask, Check, Answer and Audit. These modules contain a variety of resources that can help you tailor data cards to your organization’s needs;
- 18 Foundations. extensible frameworks that anyone can use on any type of database
- 19 Patterns of transparency. an evidence-based guide to producing high-quality data cards at scale
- 33 Participatory activities. cross-functional workshops for teams to navigate transparency challenges
- Interactive laboratory. create interactive data cards from a specified point in the browser
The Data Cards Playbook is available as a learning path for startups, universities, and other research groups.
Software tools
Our team is successful in creating tools, toolkits, libraries, and visualizations that expand the reach and improve the understanding of ML models. One such resource is Know Your Data, which allows researchers to test model performance for various scenarios through an interactive qualitative exploration of data sets that they can use to find and correct unexpected data anomalies.
PAIR recently released a new version of the Learning Interpretability Tool (LIT) for debugging and understanding models. LIT v0.5 provides image and tabular data support, new tabular attribute attribution translators, Dive visualization for layered data retrieval, and performance improvements that allow LIT to scale up to 100k databases. You can find the release notes and code on GitHub.
PAIR also contributed to MakerSuite, a rapid prototyping tool with LLMs using rapid programming. MakerSuite builds on our earlier research on PromptMaker, which received an Honorable Mention at CHI 2022. MakerSuite lowers the barrier to prototyping ML applications by expanding the types of people who can author these prototypes and reducing the time spent prototyping models from months to minutes.
 |
| A screenshot of MakerSuite, a tool that rapidly prototypes new ML models using speed-based programming, which emerged from PAIR’s quick programming research. |
Current work
As the world of AI moves forward rapidly, PAIR is excited to continue developing new tools, research, and educational materials to help change the way people think about what they can do with AI.
For example, we recently conducted exploratory research with five designers (presented at CHI this year) that explores how people with no ML programming experience or training can use rapid programming to quickly prototype functional user interface mockups. This speed of prototyping can help inform designers how to integrate ML models into products and allows them to conduct user research earlier in the product design process.
Based on this study, PAIR researchers created PromptInfuser, a design tool plugin for creating LLM-infused mockups. The plugin introduces two new LLM interactions: input-output, which makes content interactive and dynamic, and frame switching, which directs users to different frames depending on their natural language input. The result is more tightly integrated UI and ML prototyping, all within a single interface.
Recent developments in artificial intelligence represent a significant change in how easy it is for researchers to adapt and control models for their research goals and objectives. These capabilities change the way we think about interacting with AI, and they create many new opportunities for research. community. PAIR is excited about how we can use these capabilities to make AI easier to use for more people.
Gratitude
Thanks to everyone at PAIR, Rina Jana and all our partners.