Keeping same log events once instead of multiple time

Elastic Stack Logstash
1

Hello,
I have a running ELK stack, and I am collecting the logs from various source using beats and parsing them using Logstash and finally visualizing in Kibana through elasticsearch. It is working fine. I have one query,
In Kibana if we have same log event multiple times at different timestamp, can we keep these same log events once instead of multiple time?
Suppose we have some "authentication failure" log event 10 times at different timestamp, in this case my question is can we keep this "authentication failure" log event once with all the different timestamps at which it occurred?
If it is possible, for this do I need to apply any grok in Logstash or any filter in Kibana or something else?
Any help will be appreciable.
Thanks

2

If you only want to keep one copy of an event where certain fields are the same you could use a fingerprint filter to hash those fields and use the result as the document_id in elasticsearch. The document will keep getting overwritten with the most recent event. See here for an example, although I would not recommend using SHA1 for anything these days.

3

Thanks @Badger for your reply.
I am having another question,
when we talk about "Event Correlation" in the context of log Events in ELK, what does it actually mean ? Can anybody please explain this.
There is an option "SIEM" in ELK, here I found something about Event Management. I think SIEM is related to Security, we can secure our various logs in ELK using SIEM and prevent unauthorized access. But in general when we talk about "Event Correlation", can SIEM help us in this case?
Thank you.

4

No, SIEM is not security related in the sense that it controls access to logs in the ELK stack (x-pack would be used for that). SIEM is security related in that it allows you to analyse system logs to detect security threats.

5

Thanks @Badger for your reply. I got your point, I was wrong.

Actually to be specific I want to know about the "Event Correlation" in the context of log events in ELK. I am searching about "Event Correlation", but not getting anything except the concept of "Event Management" in SIEM.
In SIEM, apart from finding "security threat by analyzing the system logs" can it help us in "Event Correlation" in any way? Any suggestion on this will be very much helpful.

Thank you.

6

What do you mean by "Event Correlation"?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.