SIEM detection signals not showing up

Elastic Security SIEM
1

I am trying to run detections in 7.8 on historical data by changing the time stamps to be recent and reindexing. The data is ECS compliant. The index is in the siem defaultIndex. I have the Additional look-back time set to make sure the events are covered. Despite this, I am not getting any signals to show up. The queries work and return data if I run them in Discover.

2

Hi Darren_G - welcome to the forums!

There's a few things we can try to check to see what might be happening.

  1. Do you have access to Kibana logs and kibana.yml? If so, you can add the following line to that file in order to get more detailed logs. These verbose logs can show you more in depth information on what is happening when your rules run, generation of signals, etc.
logging.verbose: true

  1. What does your mapping look like for @timestamp? Looking back at some forum posts, there can sometimes be some funkiness depending on the mapping for this field.

  2. If you go to your rule details and select the Failure History tab, does anything show up there?

Hopefully one of these paths can help us figure out what is happening!

Best,
Yara

3

Hey Yara,

  1. I do not have access to the Kibana logs / kibana.yml

  2. I read a couple of the other posts too. I think the mapping for @timestam looks ok:
    @timestamp" : {
    "type" : "date"
    },

  3. The rules rules did not fail at all.

4

Yea, that looks ok. Well it's a good sign if you're not seeing any errors in that Failure History tab! What status is shown in the rule details page for the rule in question?

Screen Shot 2020-07-17 at 4.16.39 PM
Screen Shot 2020-07-17 at 4.16.39 PM1392×442 31.1 KB

If the rule shows succeeded and no errors are showing up as you mentioned, then the rule itself is likely running fine, but may instead be an issue with the index pattern.

Are you a part of our Slack community workspace? If you don't mind, feel free to private message me your index patterns to see if we can spot anything.

Best,
Yara

5

@Darren_G, Do the same events show up using timeline within the same time period and query?

6

The query is saying that it succeeded. I don't know how the detection query is running on the first attempt and how it uses the look-back time, so I might need a way to see the logs. I'll look into it a little more and will DM you in slack.

7

Yes, the event show up when I run the query in the timeline

8

I ended up changing the @timestamp with the date filter plugin to make it look like the data is coming in live. The test signals I wrote ended up showing up.

9

Great!

In the next upcoming release we have added a timestamp override btw so you can choose a different timestamp to represent the ingest time rather than always having to change the @timestamp.

Details are here:

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.