Open-source ML/AI models on platforms like Hugging Face can host unsafe or malicious content, with real-world demonstrations of how a model can execute code, spawn shells, or load dangerous data. The article showcases multiple attack vectors, platform safeguards, and the broader need for coordinated security across researchers, vendors, and users. #HuggingFace #AvanModel
Keypoints
- Open-source ML/AI models on Hugging Face can be unsafe or malicious, not just benign research artifacts.
- A “totally-harmless-model” used a pickle file to trigger arbitrary code on load, illustrating deserialization risks.
- Pickle-based payloads can launch external actions (e.g., opening a website) when loaded by a model.
- Hugging Face has taken proactive steps, including antivirus scans and warnings for pickle-dependent repositories, and adopting Safetensors by default.
- Models like “AvanModel” have demonstrated bind shells on port 4444, enabling remote command access.
- Other examples include a PDF with XSS payloads and a reverse-shell campaign tying to a specific IP, highlighting broader data- and content-based risks.
- The article calls for cross-disciplinary collaboration to address ethical, data-quality, and security challenges in AI/ML models.
MITRE Techniques
- [T1203] Exploitation for Client Execution – “it is possible to construct malicious pickle data which will execute arbitrary code during unpickling.”
- [T1059.004] Unix Shell – “bind shell on port 4444, thereby opening up a bind shell for an external user, or an attacker to connect to, start running arbitrary OS commands.”
- [T1059.007] JavaScript – “Cross-Site Scripting (XSS) payload that accesses your cookies.”
Indicators of Compromise
- [Domain] hosting domain – ykilcher.com
- [IP] Command-and-control / reverse shell target – 136.243.156.104
- [File] data.pkl – contains the pickle payload used to trigger code execution
- [File] result.pdf – carries an XSS payload that accesses cookies
Read more: https://blog.sonatype.com/open-source-ml/ai-models-attackers-next-potential-target