Hi there,
This is the situation:
During a cybersecurity incident we use Elastic Agent to send Windows Events to Elastic Cloud. We send all the past events from that host that are available.
As the detection rules run "in real time" they don't generate matches/alerts for past events, unless the rule has a, for example, 900 hours Additional look-back time configured.
What I would like to do:
Run all my detection rules to past events that I have just uploaded. With this once I upload all the events, even if they are from 6 months ago, it will generate alerts that will help me to do the investigation.
Your question is a common one whenever event data might be delayed before being ingested into Elasticsearch for analysis by Elastic Security detection rules.
The good news is that Elastic detection rules have an advanced setting called timestamp_override which allows the specification of an alternate timestamp to be used for rule executions. When this value is set to ECS-defined event.ingested and your integration's ingest node pipeline fills event.ingested with the time that the data was ingested, your existing rules should trigger properly without the need to modify the additional look-back time.
I just checked, and almost 300 of our current prebuilt detection rules tagged as "windows" include the timestamp_override:event.ingested setting by default, and our endpoint security and other integrations for Elastic Agent populate the event.ingested field already, so the basic scenario you describe should just work out of the box. You may have certain rules that do not include the timestamp override yet, and you should be able to modify them to also work in this scenario.
In order to find out why your rules are not triggering, can you share some additional information with us?
Which Elastic Agent integration you're using?
And what Elastic Stack version you're running?
Also, are you using Elastic Prebuilt Rules or custom rules?
Perhaps you could export and share a rule that is not triggering on the delayed data
Works great!
I just exported all my custom rules and create a small powershell script to add that timestamp_override:event.ingested to all my rules, then uploaded them and there it is...!
Thanks!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
