A lot has been written about AI and much more is awaited. However, the reason I am writing on this is much more than it being my first blog since I published “The Cyber Stack” on recent Independence Day.
I have been holding lot of training and guidance classes for PS level officers on Cybercrime investigations in my jurisdiction. The scale with which cybercrime investigation load is increasing, the day is not far when use of AI Investigators will be the compulsion and not the option.
While there is non-availability of AI-Police-Investigation-Agents, a lot is being talked about AI agents for their defending and attacking capabilities in cyber world. The matter has gone to the extent; Anthropic CEO recently called for a “slow down” with the call endorsed by OpenAI. A lot of essay competitions must have been conducted over boon and banes of AI, but the appeals for such breaks on development of AI is something new. That too coming from the very people, who have taken AI to the scale where it is today. Recently, on 10th September, Anthropic had released an R&D report namely “Detecting and countering misuse of AI”.
The report has some interesting vocabulary. GTG meaning “Generative Threat Group” is “Anthropic’s internal designators for actors observed to be abusing AI.” “Uplift” word is used at several places. It basically describes the AI capability boost, or how much more harm was caused with AI versus without AI. How the adoption of AI has “uplifted” the ability of threat actors to perform operations by scale, speed and depth. Major misuse of AI has been detected in executing cyber-attacks.
The report uses a term “Vibe Hacking”. It resembles the much popular phenomenon of “Vibe Coding” – which is basically use of natural language prompts to write codes and fix it without actually writing the code manually. Similarly, Vibe Hacking is where hackers direct AI to do all the work of reconnaissance and scanning the targets and launching the payloads until the target is achieved without using Command Line Interface (CLI) on their own. In many cases, many actors may not even be knowing the target environment also.
The report discusses in detail operations of the group codenamed “GTG-10007” establishing “Exploit Foundries”. Foundry is generally associated with “an establishment for producing castings in molten metal.” GTG-10007 had tasked Claude to construct autonomous workflows to carry out round the clock vulnerability and exploit research. The new exploits would be deployed for cyber-attacks. The actors involved in targeted roughly fifty organizations, spanning education, retail, energy, technology, healthcare, finance, manufacturing, as well as multiple government agencies globally.
You might have heard of SaaS (Software as a Service) or PaaS (Platform as a Service), but Influence-as-a-service is really very novel in the sense that you pay someone to influence someone mainly using social media. Social media has become a very profound medium to influence the public opinion these days mainly used by celebrities and interest groups which do not access to traditional print and electronic media. People prefer to get hooked to social media to consume the content they are interested in and not what someone wants to show. This has created whole new market of social media influencers, who create content for their “followers” and make money. The things turn problematic, when the content being served is fake or misleading, and people tend to believe into it because of virality etc before it is fact-checked or rebutted, but by that time the damage is done. Anthropic report defines “influence operations, …. as efforts to manipulate the information environment—including political, civic, and public discourse—with the intent to deceive, distort, or covertly influence the perceptions, beliefs, or behaviours of individuals or groups, typically while concealing the activity’s origin, sponsorship, or coordination. “
In this type of “Influence Operation”, Influence is sold as service to multiple entities especially political etc and many times it is done near to elections. The payment is done to create fake content, false social media identities and sometimes misleading news websites. The content is served so that the attribution is difficult and it appears that message is coming from some independent voice, a phenomenon described as “Attribution Laundering”. The similar influence is generated during war times and described by the actors launching the operations as “Soft War” or “Cognitive Warfare” to influence the public at home or abroad.
Another disturbing application of Claude is for surveillance purpose. Anthropic reports terms it commercial “Surveillance-for-hire” market. The AI is being used by some actors for building surveillance software that enable intelligence gathering. Claude was used also to ingest the bulk data for ex thousands of social media posts and to identify the profiles which are posting “sensitive” content. One of the GTG group, it mentions built a malicious Firefox extension using AI, meant to mass-harvest user identities from social media platforms.
The misuse does not stop at the computer screen. The report describes Claude being used for weapons software, including guidance and targeting. In one Yemen-based case, a guided rocket was test-fired, apparently unsuccessfully; Anthropic found no evidence of an operational device being fielded. This is another example of “Uplift”, where software assistance may help an actor move further with the hardware and expertise already available to him.
Biological weapons bring another dimension to this problem. Anthropic presents five cases of potential biological misuse, while expressly avoiding any assertion that the researchers intended harm. Biology is “dual use”: work useful for medicine can also support harmful purposes. For an investigator, this raises a familiar difficulty. The individual act may appear innocent, while its purpose becomes clearer only when the surrounding activities are examined together.
Anthropic says it banned accounts, strengthened safeguards and shared findings with relevant partners. But in one biological case, a platform operator regained access within days. Some weapons actors also divided work across sessions to conceal their purpose. These examples show why blocking a particular account cannot by itself settle the problem.
This brings us to “Guardrails” in AI. The word can be understood from the barriers placed along a road to prevent a vehicle from going into a dangerous area. In AI, the purpose is to restrict harmful assistance and detect misuse. But the difficulty is in understanding where the road is leading. A request to write a small piece of code may look ordinary. Its significance may change when it is connected with several other requests, a particular target and the intention of the person making them.
The warning by Anthropic CEO Dario Amodei gives a larger context to these concerns. He has expressed apprehension that, within six to twelve months, a sufficiently capable swarm of rogue AI agents could take over the internet. This is a warning about a possible future capability, rather than a claim that such a takeover has already happened.
Leave a Reply