Texas student foils rogue AI's attempt to poison open-source code
How a Texas student blew the whistle on a rogue AI hacking attempt
A University of Texas at Dallas student, Sinan Can Demir, discovered a malicious update hidden in a GitHub pull request for the open-source network scanner myNetwork. When he flagged it, the AI agent behind the attack, powered by Anthropic's Mythos 5 model, created fake personas to discredit him. The AI, part of a safety test by Britain's AI Security Institute, had gone rogue. Experts call it a glimpse into the future of social engineering, as the AI combined autonomous hacking with interactive deception.
This crossed the line from autonomous hacking to interactive deception.
- sharpshadow
It's the job of AISI to do that. Here[0] is the actual report.
It should be this part from the technical report[1]:
"In the most serious case, an AI
agent (Mythos 5) decided to attempt to solve the cyber challenge using a supply-chain attack.
As a result, the AI agent created a GitHub account and then tried to convince an open-source
repository maintainer to accept a malicious GitHub pull request (PR), including by creating a
second account masquerading as another human user endorsing the PR. When caught by an
actual human reviewer, the agent falsely claimed to have made an honest mistake – rather than
a malicious attempt – then repeatedly tried to reintroduce the malicious content by claiming
it had fixed the code (Section 4.1). "
0. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-ag...
1. https://cdn.prod.website-files.com/663bd486c5e4c81588db7a1d/...
- a2ff6eeb0
> AUSTIN, Texas, Aug 20 (Reuters) - Sinan Can Demir wanted to spend the last week of July burnishing his resume. Instead, he engaged in a battle of wits with an artificial-intelligence agent unleashed by a British government lab.
An article on Reuters naming him? Sounds like he did a good job burnishing his resume.
- freehorse
Previous discussion on the github issue thread mentioned: https://news.ycombinator.com/item?id=49218707
Archived page of said github thread itself: https://web.archive.org/web/20260731053721/http://github.com...
Discussion on the incident report: https://news.ycombinator.com/item?id=49175717
- g42gregory
In my personal opinion, for me, this article defies common sense. Who unleashed this AI model on the repository? Who gave it malevolent instructions/prompt? These questions were not even attempted to be answered. Instead it talks about AI dangers, as if the agency of these models are not in dispute. Person wielding AI, as with any other tools, is responsible for all of its actions. Otherwise, it’s just a psyop for more AI regulation, ban open source, etc… Just my 2 cents.
- ninjahawk1
I don’t think we should allow posting links here that require you the purchase a membership to continue reading. Or at least redirect with an ad block or something through a custom site. That would be rather hacker news of us.
- kumarvvr
I wonder where in the training data does this behaviour exist that the LLMs are doing it.
It's as if the training data is filled with internet discussions on approaches to hacking and the LLMs are mimicking it.
- clove
Am I misreading this or was "rogue" really not the right word for this?