secops-es-benchmark: an open benchmark for AI agents that investigate breaches in Elasticsearch
Real, labeled attack telemetry in Elasticsearch — measure your own agent against ground truth.
Today I'm open-sourcing secops-es-benchmark — the first open benchmark for SecOps investigation agents built natively on Elasticsearch and ECS.
It closes a real gap. A huge share of the world's security telemetry lives in Elasticsearch, yet there's been no public, labeled way to measure whether an AI agent can actually investigate in that environment. The datasets out there are either synthetic-and-labeled or real-but-unlabeled — neither lets you score an agent on the real task. This one does.
What you get:
- ~239k ECS documents across 12+ Elastic data streams — Elastic Endpoint, Zeek, nginx, Suricata, and detection-engine alerts.
- 5 real, non-destructive intrusions forming one end-to-end kill chain, mapped to 27 ATT&CK techniques — layered on genuine production noise, so over-alerting gets punished.
- A two-tier exam: 54 deterministically-graded questions (precise extraction) plus 5 open-ended investigations scored by an evidence-grounded LLM judge.
- A one-command runner and a live, read-only Elasticsearch + Kibana demo — point your agent at it and get a score with zero setup.
You hand an agent a single alert. It writes ES|QL, pivots across the data, reconstructs the intrusion, and recommends a proportionate response — then it's scored against ground truth.
Works with Claude or any OpenAI-compatible model. Open source, reproducible, and ready to run against your own agent today.
