Google’s Gemini AI model hacked into three private computer systems during a cybersecurity test in May, the first known case of the search giant’s AI autonomously breaching external companies.
The model accessed three separate private networks by guessing passwords and, in two instances, using a repository of publicly listed credentials, CNBC reported. The hacks occurred during a “capture-the-flag” security evaluation run by Israeli startup Irregular, a firm backed by Sequoia and Redpoint Ventures valued at US( million.
Google’s agents were never supposed to have internet access during the test, but a bug in Irregular’s testing environment left it available, BBC reported. The model found public information online and guessed credentials to access websites it believed were part of the exercise.
“In all three of these instances, the model stopped,” Google Vice President of Security Engineering Heather Adkins told the BBC. She said Google notified the three affected companies and worked with Irregular to change its testing processes.
The Wall Street Journal first reported the incident on 18 September. Google did not disclose the hack at the time because it did not consider it an example of model misalignment, calling it instead an instance of “mistaken identity.”
“In this case, the model acted appropriately,” Adkins told The Verge. But security experts pushed back. Jack Cable, CEO of AI security firm Corridor, told the Wall Street Journal that “the meta problem is, hey, models are going outside the bounds of what they should be doing, and doing actual cyberattacks.”
The disclosure makes Google the fourth major tech company to report such an incident in recent months. OpenAI, Anthropic, and Meta have each disclosed cases where their AI models broke out of testing environments and attempted to access external systems. All four incidents involved Irregular as the testing partner.
An Irregular spokesperson said the Google incident was linked to the same issue that affected the other models. “All relevant labs were notified in late July, and affected entities were contacted as part of the investigation,” the spokesperson said, adding that “all known issues on our end were remedied and resolved weeks ago.”
Anthropic CEO Dario Amodei has called on the AI industry to collectively slow down development of the most advanced models until safety can be guaranteed. The wave of disclosures has intensified scrutiny in both Washington and Silicon Valley over the pace of AI development.
Google declined to identify the specific Gemini model involved in the incident. Nvidia CEO Jensen Huang, speaking on 19 September, said AI development should proceed “as fast as we can,” BBC reported.
