Group unknown/suspicious logs that don't match a set of specific patterns - for daily analysis

Elastic Stack Kibana
1

Hello,

I'm new to the Elastic Stack and I'm trying to better understand how the software works to evaluate the performance of the open source version for my use case. My goal is to compare its functionalities against other open source and also paid softwares that are meant to receiving and analyzing logs, as well as sending alerts and reports.

The project I'm working on requires the reception of Syslog UDP logs from a variety of sources, such as linux servers and network equipment, to later classify them into two groups:

  1. the ones which we already know the format and meaning and thus don't need to be manually analyzed;
  2. the ones that are unknown for the system administrators and should be gathered in a group to be daily analyzed by the administrators. These are the potentially suspicious logs that may or may not reflect server malfunctions, attacks, etc.

In order to achieve the second grouping, our idea is to apply to the logs some kind of "negative filter", in a way that the logs that match the well-known syntax conditions for the group 1 do not show up in the group 2. What I'm looking for is the best way to apply this sort of negative filter.

It would be ideal to have a place to see all of the unknown/suspicious logs (group 1) in a daily basis. It could simply be in Kibana's Discover tab, or through a daily report, but I don't know if the Elastic Stack has this feature in open source.

So at the end, my main question is: would this idea be possible in the Elastic Stack? Is this the best way of achieving my goal? I'm always open to suggestions!

About the new Alerting feature in version 7.7, I'm glad that it has been released and I updated from 7.6.2 just to test it out, but I was let down by the fact that all the useful alerting methods, such as e-mail, are available only with a paid subscription. It would be a huge advantage for the Elastic Stack to have a native alerting feature in open source, mainly to rival some open source and paid competitors, but it seems like there's no such feature yet. For my alerts, I'm trying to use Elastalert.

I'd like to thank you in advance,

Felipe Silveira

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.