Keypoints
- AI-generated deepfakes (audio, video, images) were used in multiple recent national and local elections to promote candidates or discredit opponents.
- Audio deepfakes are more frequently and successfully used than video/image deepfakes because the technology is mature and easier to deploy and hide.
- Tools and services for creating deepfakes are widely available, from GitHub repositories (3,000+) to commercial platforms and criminal-market offerings.
- Campaigns used automated networks of fake social media accounts and platform-specific services to post and amplify AI-generated content at scale.
- Disinformation content is often released shortly before voting or during media blackout periods to limit verification and rebuttal by fact-checkers.
- Commercial AI video services (e.g., HeyGen, D-ID) and advanced voice-conversion systems (RVC) were explicitly cited as sources for fabricated materials.
- Many countries lack specific laws addressing AI-fabricated media, complicating legal and platform-level enforcement against misuse.
MITRE Techniques
- [T1098] Account Manipulation – Creating and operating large numbers of fake or automated social media profiles to distribute content (‘…overseeing hundreds of accounts, facilitating daily posts…’)
- [T1566] Phishing – Using AI-generated spam and targeted messages to increase influence or deliver disinformation (‘…AI-powered spam emails have surfaced…increase their success rates in targeting individuals.’)
- [T1583] Acquire Infrastructure – Procuring or renting tools and services (criminal-market platforms, commercial AI services, GitHub repos) to generate and host deepfakes (‘…over 3,000 GitHub repositories are dedicated to the development and dissemination of deepfake technology…’)
- [T1102] Web Service – Leveraging third-party AI video/voice services (HeyGen, D-ID, Sora) and mainstream platforms for content generation and hosting (‘…Fake videos were reportedly generated using HeyGen…and D-ID…’)
- [T1204] User Execution – Timing the release of convincing fabricated media shortly before elections to reduce time for verification and provoke user reaction (‘…often disseminated shortly before election dates to limit the opportunity for fact-checkers to respond.’)
Indicators of Compromise
- [Service / Tool] Platforms used to generate deepfakes – HeyGen, D-ID (commercial AI video services) and OpenAI Sora (text-to-video capability).
- [Repository count] Open-source tool presence – “over 3,000 GitHub repositories” for deepfake technology (indicates broad availability of code and models).
- [Social account] Amplifier accounts – Instagram account ‘iaxlapatria’ used to distribute AI-generated images in Argentina; other anonymous profiles reposting fabricated content.
- [Content artifact] eBook/source seed – “The Secret History of Tsai Ing-wen” eBook served as source material amplified into AI videos and avatars.
- [Platform hosting] Distribution channels – YouTube and TikTok videos (e.g., AI-generated victory speech on YouTube), and social platforms like X (formerly Twitter) used to spread materials.
AI deepfakes are created using a mix of commercial services (HeyGen, D-ID), voice-conversion systems (RVC-style models), and open-source model code available on GitHub. Operators prepare source material—text prompts, existing audio/video clips, or synthesized voice prints—and feed them to generation pipelines to produce polished audio, video, or image artifacts. Voice cloning workflows commonly rely on small voice samples to produce convincing speech matching a target, while image/video pipelines use generative diffusion or GAN-based models and text-to-video tools (e.g., Sora) to animate or alter faces and scenes. These generation steps can include post-processing for noise reduction, lip-sync alignment, and format conversion to match platform norms.
For distribution, actors employ bought or built infrastructure: marketplace offerings and underground platforms provide automated posting tools that manage hundreds of fake profiles, schedule daily posts, and coordinate cross-platform propagation. Content is often pushed through social media, instant messaging (personalized WhatsApp messages), email spam campaigns, and robocalls to reach specific demographics. Operators exploit timing—releasing artifacts shortly before elections or during media blackout periods—to reduce verification windows and maximize impact. Amplification is further achieved by leveraging sympathetic influencers, repost networks, and paid placement to elevate reach.
Detection and mitigation are hampered by the accessibility and maturity of audio-manipulation tools, the volume of available code and services, and gaps in regulation. Audio deepfakes are particularly challenging due to limited contextual cues and rapid production cycles; automated analysis can flag anomalies but often requires cross-platform intelligence and rapid response. Effective mitigation focuses on reducing infrastructure acquisition (platform and marketplace takedowns), hardening distribution channels (spam/robocall rules, account-verification), improving media provenance and watermarking, and accelerating fact-check workflows to counter last-minute releases.