Agents vs. agents: how we triage HackerOne reports for $2 each, 85% as well as a human

Agents vs. agents: how we triage HackerOne reports for  each, 85% as well as a human
Elastic built an AI-assisted bug bounty triage system to handle a surge of AI-generated HackerOne reports, using multi-stage analysis, adversarial review, and sandboxed reproduction on ephemeral Google Cloud VMs. The system agrees with human security engineers 85% of the time and uses Elastic-specific triage rules to distinguish real issues from features or out-of-scope reports while keeping a human in the final decision loop. #HackerOne #Elastic #Claude #ElasticWorkflows #GoogleCloudPlatform #Elasticsearch #Kibana

Keypoints

  • Elastic saw over 1,390 bug bounty reports in the first half of 2026, exceeding the combined totals from 2024 and 2025.
  • The triage system was designed to reduce the cost of handling near-zero-cost AI-generated reports by using AI for first-pass analysis.
  • The pipeline uses two isolated ephemeral VMs: one for analysis and one for sandboxed reproduction when needed.
  • Retrospective validation against 764 known-outcome reports showed 85% agreement with human security engineers.
  • An adversarial review stage independently challenges the analysis and can route disagreements directly to human review.
  • Elastic calibrated the system against more than 3,300 historical reports to encode product-specific triage rules and category thresholds.
  • Human reviewers still make the final call on every report, including any external action such as responding to researchers or creating GitHub issues.

MITRE Techniques

  • [T1190 ] Exploit Public-Facing Application – Vulnerability reports target externally reachable products and services during triage and reproduction (‘an external party in an environment you control’).
  • [T1189 ] Drive-by Compromise – Report text is treated as untrusted input that can attempt prompt injection or manipulation without special access (‘prompt injection via report text’).
  • [T1059 ] Command and Scripting Interpreter – Reproduction steps may execute attacker-described commands inside the tester container (‘executes researcher-described steps inside a sandboxed tester container’).
  • [T1609 ] Container Administration Command – The system explicitly runs Docker Compose and containers to provision reproducible environments (‘provisions an Elastic Stack via Docker Compose’).
  • [T1611 ] Escape to Host – The threat model includes attempts to break out of the Docker container to the host VM (‘Sandbox escape: Reproduction steps that attempt to break out of the Docker container to the host VM’).
  • [T1021 ] Remote Services – Reproduction environments and product containers communicate over isolated networks while testing remote services (‘the tester can reach them’).
  • [T1041 ] Exfiltration Over C2 Channel – The system considers credential and data exfiltration from analysis or reproduction environments (‘attempt to extract API keys, cloud metadata, or internal configuration’).
  • [T1496 ] Resource Hijacking – Resource abuse scenarios include cryptocurrency mining and expensive cloud provisioning (‘cryptocurrency mining’).
  • [T1499 ] Endpoint Denial of Service – The triage rules discuss DoS reports, including out-of-memory crashes and volumetric abuse (‘a single crafted request that causes an out-of-memory crash’).
  • [T1053 ] Scheduled Task/Job – Workflow orchestration triggers analysis and reproduction automatically on new reports (‘An Elasticsearch alert rule triggers the workflow when a new report appears’).

Indicators of Compromise

  • [IP address ] egress proxy for outbound traffic from triage VMs – 10.0.0.10:3128
  • [Cloud service / infrastructure ] ephemeral analysis and reproduction hosts – e2-standard-2, e2-standard-4, and other VM types
  • [File path ] baked agent location inside the VM image – /opt/vuln-triage-agent
  • [Cloud service / storage ] results uploaded from each triage run – Google Cloud Storage (GCS)
  • [Cloud service / secrets ] credentials injected only when needed – Google Cloud Secret Manager, and other secret-store items
  • [Domain / platform ] bug bounty intake and triage platform – HackerOne
  • [Product / service ] internal search and alerting store used as the triage trigger source – Elasticsearch
  • [Product / service ] sandboxed reproduction stack components – Kibana, Logstash, Beats, APM Server, and other 2 topology items
  • [File / deployment artifact ] reproducible environment definition – Docker Compose, Packer-built image, and Jinja2-rendered templates


Read more: https://www.elastic.co/security-labs/ai-vulnerability-triage-bug-bounty-hackerone