# Replay rule on old index dates **URL:** \ **Category:** Elastic Security\ **Created:** [May 12, 2021, 9:57pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857 "2021-05-12T21:57:55Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 12, 2021, 9:57pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/1 "2021-05-12T21:57:55Z")
Hi there, I have 90 days worth of windows event logs (partitioned per day) and a set of rules that I want to apply on those days. I noticed that when I activate a rule it can only go forward in time, is there a way to effectively replay the rule on past historical events to generate the alerts on each of the days? This is for a SOC blue team exercise to monitor the team performance when triaging events. The only option I see is to sort of copy and paste old events as if they were new, but this will be a massive job on the order of TB size. Any help is appreciated! ---
**Author:** ![madi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madi/32/75699_2.png) [@madi](https://discuss.elastic.co/u/madi)\ **Post date:** [May 13, 2021, 4:43pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/2 "2021-05-13T16:43:19Z")
Hi @priamai! Thanks for your question. There are 2 options I can think of... 1. set a really long look-back interval on the rule ... this could be problematic for a number of reasons and won't work for all rule types, but it could be a possibility. You could easily run into performance problems and/or hit your maxSignals limit. 2. use a timestamp override. If your data has `event.ingested` and it was imported recently, that might be an option. Otherwise, if you don't have another more recent timestamp available on the events, you may need to take the approach from your original question (rewrite the events with a more recent timestamp, either using the `@timestamp` field or a new override field). There has been talk about adding more explicit support for this type of thing, but as far as I know, it hasn't yet been prioritized. In the meantime, one of the above may have to suffice (I wonder if runtime fields could be another option, combined with a timestamp override?) Hope this helps! Madi ---
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 13, 2021, 8:37pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/3 "2021-05-13T20:37:08Z")
I just thought of a third option, let me know if this is possible: 1. run the rule queries with my client code 2. emulate the output of the alert in the correct index I could not find any information about how ES manages the indexes related to the Detection engine. Are there any documents I could look into? Otherwise I can generate a few events with your approach such as (2) and then figure out the index alert structure? Cheers. ---
**Author:** ![austinsonger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/austinsonger/32/78994_2.png) [@austinsonger](https://discuss.elastic.co/u/austinsonger)\ **Post date:** [May 14, 2021, 5:20pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/4 "2021-05-14T17:20:08Z")
Why don't you just perform Adversary emulation and generate fresh detections? [redcanaryco/atomic-red-team: Small and highly portable detection tests based on MITRE's ATT&CK. (github.com)](https://github.com/redcanaryco/atomic-red-team) ---
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 15, 2021, 8:45am UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/5 "2021-05-15T08:45:08Z")
Yes we do that already (with redcanary and other commercial BAS) for training purposes of new SOC analysts, but for this particular exercise we do have already historical data with known incidents that we want replay together with baseline (attack free) events. ---
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 15, 2021, 8:49am UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/6 "2021-05-15T08:49:25Z")
So I have been doing some experiments and there is indeed an hidden index called: .siem-signals-default-\*\*\*\*\* which contains the the event that triggered and the rule relationships in the signal document. It seems not a big deal to write an injector for that but any documentation on that index is welcome. Cheers. ---
**Author:** ![austinsonger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/austinsonger/32/78994_2.png) [@austinsonger](https://discuss.elastic.co/u/austinsonger)\ **Post date:** [May 15, 2021, 10:39pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/7 "2021-05-15T22:39:34Z")
If you really need too then: 1. Stop filebeat and logstash. 2. Clear old data from Elasticsearch if there. 3. Delete registry files `registry` and `registry.old` . 4. Run logstash. 5. Run filebeat using command `filebeat -e -once` . ---
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 16, 2021, 6:16am UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/8 "2021-05-16T06:16:35Z")
Yeah but for the historical data is more than 5TB of streamed logs and we don't keep the original evt files or similar for filebeat to replay. ---
**Author:** ![priamai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/priamai/32/80269_2.png) [@priamai](https://discuss.elastic.co/u/priamai)\ **Post date:** [May 16, 2021, 6:20am UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/9 "2021-05-16T06:20:00Z")
I managed to butcher a solution and generate an alert directly into the index, is working but not elegant. ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/3/83b1139d497a3eed4aacd214c19c936d69be39e0.png) What I discovered is that the rule object nested inside the alert is not identical to the rule object you get from the kibana api. ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/5/6584421ded9d3a37135929912b212b528fab2c7e.png) The one from the API has very few fields compared to what is inside the siem index. For now I do like this: 1. for each rule 2. generate 1 trigger event from the past 3. copy the rule object from the test event 4. add the new event information ---
**Author:** ![madi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madi/32/75699_2.png) [@madi](https://discuss.elastic.co/u/madi)\ **Post date:** [May 18, 2021, 2:56pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/10 "2021-05-18T14:56:34Z")
@priamai Which APIs are you using? Can you just use the source data from the index? Is this a workable solution for you in the meantime? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 15, 2021, 2:57pm UTC](https://discuss.elastic.co/t/replay-rule-on-old-index-dates/272857/11 "2021-06-15T14:57:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.