Google details adversarial AI activity on Gemini

State-sponsored threat actors are using Google's Gemini generative AI assistant to perform security research, assist with coding, and create content to influence online audiences, according to new research from the tech giant.

Google on Wednesday published "Adversarial Misuse of Generative AI," a research blog documenting the Google Threat Intelligence Group's (GTIG) analysis of how threat actors attempted to misuse the Google Gemini chatbot as well as other publicly available generative AI tools. A Google spokesperson told Informa TechTarget that the research was conducted over the course of 2024.

In particular, the report focuses on adversarial AI activity from government-backed advanced persistent threat (APT) as well as coordinated information operations (IO) actors. Google defines the latter as threat actors that "attempt to influence online audiences in a deceptive, coordinated manner" via means including sock puppet accounts.

Much of Wednesday's research is centered around how threat actors unsuccessfully tried to exploit Gemini for significant gain.

For example, GTIG researchers said threat actors attempted and failed "to use Gemini to enable abuse of Google products, including researching techniques for Gmail phishing, stealing data, coding a Chrome infostealer, and bypassing Google's account verification methods." Additionally, threat actors used basic and publicly available jailbreak prompts to unsuccessfully bypass Gemini safety controls.

Additionally, GTIG researchers noted that they did not observe "any original or persistent attempts by threat actors to use prompt attacks or other machine learning (ML)-focused threats as outlined in the Secure AI Framework (SAIF) risk taxonomy." The SAIF categorizes prominent adversarial AI threats such as data poisoning, model exfiltration and prompt injections, which are attempts to bypass LLM restrictions by hiding inputs within seemingly legitimate prompts.

Google said Gemini was used for "several phases" of the APT attack lifecycle, such as researching relevant targets, tools and vulnerabilities; troubleshooting code; developing payloads; and assisting "with malicious scripting and evasion techniques." On the IO side, Gemini was used for research, content generation "including developing personas and messaging; translation and localization; and to find ways to increase their reach."

"Rather than enabling disruptive change, generative AI allows threat actors to move faster and at higher volume," GTIG researchers said. "For skilled actors, generative AI tools provide a helpful framework, similar to the use of Metasploit or Cobalt Strike in cyber threat activity. For less skilled actors, they also provide a learning and productivity tool, enabling them to more quickly develop tools and incorporate existing techniques."

Google identified APTs from more than 20 countries that used Gemini. Iranian actors were the heaviest Gemini users for both APT and IO threat activity, using it for a wide range of aforementioned adversarial AI activity. Chinese APTs used Gemini for reconnaissance, coding development and troubleshooting, and research on how to gain deeper access into target networks.

North Korean actors used Gemini similarly to China, though they also used it to research topics such as cryptocurrency, the South Korean military and free hosting providers. Russian use was predominantly coding assistance, though Google "observed limited use of Gemini during the period of analysis."

Google claimed that although the chatbot provided assistance with explaining concepts, executing simple coding tasks or creating written content, Gemini's safety measures triggered whenever a threat actor would attempt to use the assistant for more "explicitly malicious tasks." While this led to some productivity gains, GTIG said, Gemini use did not lead to the creation of any novel threat actor capabilities.

GITG researchers noted, for example, that a Chinese nation-state APT unsuccessfully attempted to use Gemini to reverse engineer Carbon Black's endpoint detection and response product.

"Gemini did not produce malware or other content that could plausibly be used in a successful malicious campaign," the research read. "Instead, the responses consisted of safety-guided content and generally helpful, neutral advice about coding and cybersecurity. In our continuous work to protect Google and our users, we have not seen threat actors either expand their capabilities or better succeed in their efforts to bypass Google's defenses."

In a statement shared by a Google spokesperson with Informa TechTarget, GTIG researchers said they did not observe threat actors using Gemini to develop any new attack techniques.

"The activity that we observed was actors researching existing and previously published techniques related to evasion, lateral movement, and privilege escalation," GTIG said. "Several threat actors attempted to use Gemini for vulnerability research, but this usage focused on publicly reported vulnerabilities and specific CVEs."

In a separate blog post on Wednesday, Ken Walker, president of global affairs at Google & Alphabet, emphasized that threat actors haven't demonstrated that generative AI can be abused to develop new capabilities.

"In other words, the defenders are still ahead -- for now," Walker wrote. "To keep it that way, particularly as powerful new models -- which can be leveraged by a wide variety of actors -- begin to gain traction, American industry and government need to work together to support our national and economic security."

To that end, Walker called for government support and policies that help American companies maintain their lead in the AI technology market as well as expanded collaboration between the public and private sectors to strengthen cyber defenses.

Alex Delamotte, a threat researcher at Sentinel One's SentinelLabs, said GTIG's findings align with what she and other AI-focused researchers have observed regarding adversarial AI activity.

However, she added that although the report stated threat actors were unsuccessful in getting Gemini to generate explicitly malicious code, "it's worth noting that actors are readily using these models to generate code that is not inherently malicious, as outlined in the report's activity detailing People's Republic of China (PRC)-aligned actors who used Gemini to generate code to access Windows Event Logs."

"There is a huge breadth of opportunity for actors to generate code that streamlines their operations without violating the model's safety guardrails, much like red teams have always relied on open source code projects to simplify development time invested in their operations," she said.

Similarly, Sergey Shykevich, threat intelligence group manager at Check Point Software Technologies, said his company's findings align closely with Google's.

"At present, different threat actors are primarily using AI to streamline their daily activities, increasing efficiency and effectiveness," Shykevich said. "However, we still don't see creation of sophisticated or previously unseen malware. Most of the observed uses involve generating content, crafting phishing emails, developing scripts and aiding in the creation of harmful code development."

Alexander Culafi is a senior information security news writer and podcast host for Informa TechTarget.