Elastic built an evidence-based evaluation framework for agentic SOC models that tests real tool use, grounded decisions, and workflow execution inside Elastic Security rather than relying on generic LLM benchmarks. The article also covers parallel suites for Attack Discovery and Automatic Migration, all scored blind to measure quality and reliability across security tasks. #ElasticSecurity #Chrysalis #VirusTotal #Splunk #QRadar #MicrosoftSentinel
Keypoints
- Elastic argues that generic LLM leaderboards are poor proxies for agentic SOC work because they do not verify tool use or grounded results.
- The evaluation framework runs the same prompts against every model in a live Elastic Security deployment while capturing full tool traces, parameters, results, latency, and tokens.
- A synthetic Chrysalis intrusion, including a malicious loader using an EICAR test hash, seeds realistic alert data for repeatable testing.
- The agent suite evaluates seven capability categories, including alert analysis, entity analytics, threat hunting, detection rules, workflow authoring, triggering workflows, and multi-step response.
- Two additional suites assess Attack Discovery and Automatic Migration, with separate answer keys and scoring methods.
- Blind judging, anonymized models, and a “no tool call, no credit” rule are central to keeping scoring fair and evidence-driven.
- The article’s main conclusion is that model rankings vary by capability, so the best choice for an agentic SOC depends on the specific task.
MITRE Techniques
- [T1574.002] Hijack Execution Flow: DLL Side-Loading – Used in the Chrysalis scenario where a legitimate-looking executable loads a malicious DLL (‘a DLL side-loading attack on a Windows host, srv-win-defend-01, where a legitimate-looking BluetoothService.exe loads a malicious log.dll’).
- [T1055] Process Injection – Not explicitly named, but the evaluation includes hunting for suspicious process activity and malware behavior associated with the intrusion (‘look for BluetoothService.exe execution, suspicious DLL loads’).
- [T1071.001] Application Layer Protocol: Web Protocols – The framework checks for command-and-control activity through network telemetry during threat hunting (‘any C2 network or DNS activity’).
- [T1071.004] Application Layer Protocol: DNS – Hunt prompts specifically ask the model to look for DNS-based C2 indicators (‘any C2 network or DNS activity’).
- [T1016] System Network Configuration Discovery – The threat-hunting prompts include investigation of host and network artifacts to understand the intrusion (‘hunt the logs for corroborating activity’).
- [T1082] System Information Discovery – Entity and host analytics are used to investigate affected systems and users with context (‘investigate hosts and users with purpose-built entity lookups and risk context’).
- [T1497.001] Virtualization/Sandbox Evasion: System Checks – The article discusses EICAR as a harmless test string used to trigger antivirus detection, not an attacker technique, but it is used to validate detection behavior (‘EICAR is the industry-standard harmless test string that every antivirus engine flags as malicious by design’).
- [T1588.001] Obtain Capabilities: Malware – The scenario centers on a malicious loader and backdoor campaign used in the evaluation (‘the Chrysalis backdoor campaign’, ‘malicious loader’).
Indicators of Compromise
- [Hostnames] affected endpoint and incident systems – srv-win-defend-01, and one more host mentioned in the alerting dataset
- [File names] suspicious loader and side-loaded DLL – BluetoothService.exe, log.dll
- [File hashes] VirusTotal validation target and test hash – 275a021b…fd0f, EICAR test file hash
- [Tool names / integrations] response and lookup actions used in the workflow – vt.hash.lookup, create.case, create.channel, check.on.call.schedule
- [SIEM / platform sources] migration targets and sources referenced in the evaluation – Splunk, QRadar, Microsoft Sentinel, Elastic Security
- [Cloud / collaboration systems] incident coordination platform mentioned in workflow actions – Slack, and one more live connector-based system
Read more: https://www.elastic.co/security-labs/llm-benchmarking-agentic-soc