Keypoints
- Malicious LLMs (e.g., WormGPT, FraudGPT) are being used to generate phishing, BEC emails, and malware code.
- AI content detectors analyze perplexity and burstiness to score likelihood of AI-generated text, but can produce false positives.
- Common linguistic indicators of AI-written emails include formal/matter-of-fact tone, repetitive phrasing, long uniform sentence structure, minimal typos, and occasional hallucinations or outdated facts.
- Practical examples observed include payroll-diversion BECs and HR-themed phishing where multiple detectors flagged AI-generated content.
- Detection accuracy improves with longer text samples; many detectors have input-size and language-support limitations that reduce reliability for short or non-English emails.
- Researchers recommend using multiple detectors and linguistic analysis together; CopyLeaks and GPTZero performed well in the tests cited.
- Despite detection efforts, public awareness and user education remain essential since detection tools alone cannot stop AI-assisted attacks.
MITRE Techniques
- [T1566] Phishing – AI-generated and tailored spam/BEC messages are distributed via email to deceive recipients (‘we are seeing these potentially AI-written spam being distributed via email.’).
- [T1566.002] Spearphishing Link – Emails instruct recipients to click links (e.g., reset-account links) to capture credentials or deliver payloads (‘…need to click the link to reset their account’).
- [T1204] User Execution – Social-engineering content persuades users to perform actions such as updating payroll details or following recovery steps (‘asking for assistance in changing their supposed payroll accounts.’).
- [T1587.001] Develop Malware – Malicious LLMs are used to generate or assist in producing malware code and malicious pages (‘They can craft convincing Business Email Compromise (BEC) email, phishing pages, and malware codes.’).
Indicators of Compromise
- [Domain] source/analysis – https://www.trustwave.com/en-us/resources/blogs/spiderlabs-blog/the-inevitable-threat-ai-generated-email-attacks-delivered-to-mailboxes/ (original Trustwave post)
- [Malicious LLM names] models used to craft attacks – WormGPT, FraudGPT (used to create BEC/phishing content and malware code)
- [Detection tools] analysis/reporting – CopyLeaks, GPTZero, ZeroGPT, Quillbot, Sapling, Undetectable AI, Writer, Scribbr (used to evaluate sample emails; varied verdicts reported)
- [Suspicious email strings] repeated phishing phrases – “payment details for payroll”, “click the link to reset their account” (phrases observed in payroll-diversion and HR-themed phishing examples)
Generative LLMs are being leveraged to produce targeted phishing and BEC messages; attackers use refined prompts or malicious LLMs (e.g., WormGPT, FraudGPT) to craft personalized payroll-diversion and HR-themed phishing emails and to generate phishing pages or malware code. Detection focuses on statistical and linguistic analysis—detectors compute perplexity and burstiness to score AI-likelihood, while analysts look for telltale signs such as overly formal tone, repeated phrasing or sentence patterns, long uniform sentences, minimal typos, and occasional factual errors (hallucinations).
Operational detection practice requires combining multiple AI-content detectors with manual syntactic review. In the tested examples, tools like CopyLeaks, GPTZero, ZeroGPT, Quillbot, Sapling, Undetectable AI, Writer, and Scribbr produced mixed results across short and long BEC/phishing samples; longer inputs yielded more reliable detector outputs. Analysts should feed sufficiently long samples where possible, compare results across detectors, and correlate tool scores with linguistic indicators (tone shifts, POV inconsistencies, repeated clauses) rather than relying on a single verdict.
Be aware of tool limitations: many detectors have minimum input lengths, limited language support, and bias against non-native English styles—yielding false positives for short or non-native-content emails. Practical recommendations are to use multiple detectors (the research highlighted CopyLeaks and GPTZero for strong performance), perform sentence-level analysis, and prioritize user awareness/training because detection tools alone cannot fully prevent AI-assisted phishing and BEC attacks.