Pathways Language Model (PaLM)

What is Pathways Language Model (PaLM)?

The Pathways Language Model is the name of a family of AI large language models developed by Google. The effort gets its name from a Google Research initiative to create what researchers dubbed pathways, in an approach designed to build a single powerful model that could serve as a foundation for multiple use cases.

There are multiple versions of the Pathways Language Model (PaLM). Among the versions of PaLM 2 are Med-PaLM 2, which is fine-tuned for life sciences and medical information; and Sec-PaLM, which is focused for use in cybersecurity deployments to expedite threat analysis.

In May 2023, Google publicly stated that its Bard conversational AI technology is powered by PaLM 2. The PaLM 2 large language model (LLM) also enables generative AI capabilities in the Google Workspace suite of applications -- including Gmail and Docs -- as well as Google Cloud with a technology known as Duet AI.

What can PaLM do?

PaLM -- and more specifically PaLM 2 -- can provide many functions, including the following:

• Text generation. PaLM 2 generates text on any topic a user requests using a text prompt.
• Summarization. Another core capability that summarizes large volumes of content into a more compact form.
• Content analysis. This feature helps users understand what's in a given block of content. This can include sentiment analysis to identify if the tone of the content is positive or negative.
• Reasoning. An improved attribute of the PaLM 2 model is the ability to reason. PaLM 2 has a diverse data set that encompasses scientific papers and content with mathematical expressions. This data set enhances the model's proficiency in logic and common-sense reasoning about problem sets provided by users via a prompt.
• Code generation. PaLM 2 generates computer programming code in 80 different languages, including Java, JavaScript and Python.
• Code analysis. The model can look at a block of code and identify potential bugs or coding errors.
• Text translation. PaLM is trained in multiple languages and can execute text translations.

How does PaLM work?

PaLM uses a transformer neural network-based model commonly referred to as a transformer. At a basic level, PaLM is similar to rival transformer-based models, including OpenAI's GPT-3 and GPT-4 models.

PaLM uses the Google-developed Pathways machine learning system to train a model across multiple pods of tensor processing units. The model uses a technique known as few-shot learning that lets it learn from a limited number of labeled examples -- or shots -- to help it quickly adapt and generalize new tasks or classes with minimal data labeling.

As a transformer network, PaLM understands and creates patterns across content, including text and code. The transformer model undergoes a comprehensive learning process, uncovering statistical patterns and connections that exist among words and phrases within content. This acquired knowledge empowers PaLM to generate responses that are both coherent and relevant in various contexts.

What are the limitations of PaLM?

While PaLM is powerful, it has the following limitations on use and capabilities, as well as other items of concern.

• Use. PaLM is a Google-developed and published model. With the launch of PaLM 2, Google has opened some use for external developers via API, Firebase and on Colab; however, commercial terms of use aren't clear. External developers can't contribute new code or help in the development of PaLM as it's a proprietary model and not open source.
• Images. PaLM 2 can bring in visual results as part of a query. What it can't do entirely on its own is generate new images. However, Google does let tools built with PaLM 2 -- including Bard -- be extended with support. For example, Bard can connect with Adobe Firefly to let users create AI-generated images.
• Explainability. PaLM is a closed model and doesn't provide much -- if any -- details to support explainable AI, which is critical for users and organizations to understand how a model comes to a specific decision. Explainable AI is an issue of growing importance as it lets users and organizations understand models better, so they can be trusted more.
• Toxic content. A key limitation of PaLM identified by Google researchers is the risk of toxic content -- content that can be construed as being biased, malicious or harmful to users.

"Prompted dialog systems built from PaLM 2 continue to produce toxic language harms, and to exhibit patterns of bias in how those harms vary by language and queries related to identity terms," Google researchers wrote in the "PaLM 2 Technical Report."

Differences between PaLM and GPT-3 and GPT-4

There are many similarities and differences between PaLM and OpenAI's GPT-3 -- and the more recent GPT-4 -- LLMs. Both sets of technologies are generally referred to as generative AI, and they both benefit from the use of a transformer model for deep learning. Both technologies can also create, summarize and understand text.

History of PaLM

Google announced PaLM in April 2022, with the initial version of the language model providing 540 billion parameters.

Google researchers provided early metrics on PaLM performance in a research paper titled "PaLM: Scaling Language Modeling with Pathways," including full details explaining the innovations the model introduces. The model remained private until March 17, 2023, when Google provided an initial set of public APIs to let developers try the model. On May 10, 2023, PaLM 2 was publicly announced at the Google I/O conference.

At the time of the PaLM 2 launch, Google didn't publicly disclose the size of the model in terms of parameters, though it made many claims about the model being larger, more capable and overall providing better performance than its initial model.

Google did disclose that PaLM 2 is being used as the foundational AI behind many of its generative AI efforts, including Bard.

Looking forward, it's unclear if there will be a PaLM 3 model. At the Google I/O 2023 event, company executives said the next-generation LLM being developed by the organization is called Gemini. It's not clear what -- if any -- relationship Gemini has to the PaLM-based approach.