Include custom Elasticsearch index in SIEM default dashboards

Elastic Security SIEM
1

Hi people. I have an ELK server 7.8.0 running OK in testing mode.

I've configured Logstash to listen on UDP/514 for incoming syslog remote events, Logstash doesn't apply ani filter, just pass the input to Elasticsearch. I've create a new custom index called syslog-network-"YYYYMMDD".

When I go to KIbana's Discover section, I can see these type of log (it's just a sample):

imagen

When I go to Kibana's SIEM access link, and after that I go to Events section I can't see the syslog events at all (I've created a custom dashboard looking into my custom index and I can see aprox. 50 millons syslog events by day).

This is my Events panel from SIEM, and it doesn't show any syslog event at all, and syslog events are greater than netflow events:

imagen

How can I include the custom index in the events panel from SIEM section of Kibana?

Every time I create a custom index, do I have to do any special task/action ? Because I only create an index pattern for each of them, no more actions I take.

Thanks a lot and regards !!!

2

Hi there,

you can add an Index pattern in Stackmanagement-> advanced settings-> Siem default Indices

You can use wildcards. Maybe you define a prefix for all new indices which should be used in SIEM.

You should not put all Data inside SIEM. Only the indices where you wann to use detections or similar.

The Rest is still searchable in discover or normal dashboards

3

Dear Logger, ok...now I understand!!!

Please let me ask you two short questions:

  • Is it relevant the fields of the incoming logs stored in my custom index? Do they show anyway in the SIEM panels? Because I did you told me and the syslog events don't appear in the SIEM. Maybe do I have to apply a Logstash filter to incoming syslog data before send them to Elasticsearch?

This is one of my incoming logs:

imagen
imagen842×149 21 KB

  • If I have indices like syslog-YYYYMMDD and syslog-network-YYYYMMDD, in the SIEM default indices should I have to put syslog-* ? Or syslog-; syslog-network-?

Special thanks again!

4

Hi.

First.
They show up there. But without the ECS Fields extracted, it is not that helpful.
Have a look at the elastic Common schema.

Second.
You can just use syslog-*

Sorry for my short answer. But I think you can manage this

Greetings

5

Dear Malte,I appreciate your help, it's very important to me.

You tell me that the logs "show up there", but I've shown you the Discover panel and not the SIEM events panel. I don't see any syslog message on the SIEM panel now.
I'm not an expert at ELK, I'm starting in this fantastic world, so let me ask you:

Do I have to use an ECS filter in Logstash in order to see the syslog events on the SIEM panel ? Or may I use a Logstash syslog plugin instead ?

Thanks in advance again!

6

Hi Robert,

If you add the syslog-* to the settings then the logs will show up in the SIEM section in Kibana.
add_index_to_siem
This is a Screenshot from my Kibana.

The SIEM from Elastic uses the ECS. If you modify your logs that they meet the ecs then it is usable with the siem.

Good for you is that you have cisco asa firewalls. With this you should use the Filebeat https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-cisco.html
Filebeat is quite powerful and is has a module for a lot of common used systems. Logstash itself has no ECS Filter, but it is possible to just grok or dissect the message and then name the fields as needed.

If you go with Filebeat then you should have the best outcome.

Have in mind that you should first know what you want to see in the SIEM and then put the needed data inside, instead of putting data inside and then thinking about what could be done with this data.

If you dont want to use Filebeat, then you should be able to put the syslog-* pattern in the advanced settings and then you can modify the detection rules to meet your fields or queries.

I hope this will help you.

Greetings
Malte

7

Hi Malte, you helped me a lot, great explanation !

So next week I'll start to play with what you told me.

Thanks and regards!!!

8

Dear Malte, please let me ask you a last question:

Suppose I need to collect logs (syslog) from different platforms: Cisco routers and switches, Cisco ASA Firewall, generic UPS, Linux and Windows servers, VMware hosts.

What is the best solution for doing this? Maybe Cisco ASA and Cisco routers/switches sending logs to Filebeat cisco module to two different UDP ports, and the other devices sending to Logstash on another UDP port and after that passing thgem to Elasticsearch ? Or what ?

Thanks a lot again !!!!!

9

Hi Robert,

You can use for all udp/tcp/syslog source the filebeat. Windows Server should be collected by winlogbeat. It is awesome for that.

Logstash is a great tool for enrichement of data or for normalization. But if you just want to use syslog it is possible to just use filebeat.

You can also define different udp/tcp/syslog ports in filebeat. Have a look at https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-syslog.html

the inputs can be defined with different ports.
You can also put input configuration inside a config folder where it reloads the configuration files.

Fo the beginning I would recommend you start with the defaults and use the modules (https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-modules.html) and if there is no module then you use a standard input (https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html)

Then after a while you know your data and you can start building a SIEM :wink:
It takes some time but it is worth it

10

Dear Malte, you hjelp me a lot!!!

Today I'm going to read the links you sent me.

Maybe I'll ask you a specific question in the future.

Greetings !!!

11

Dear Malte, how are you?

Just a short question please:

What happens if I point my Cisco ASA to a syslog standar input from Filebeat in place to point it to the ASA option from the Cisco module of Filebeat???

Will I have the same message fields and detection capacity in SIEM? What is the advantage in using Filebeat modules in place of Filebeat syslog standard input???

Thank you very much!!!

12

Hi Robert.

I think you need a few more basics before building a SIEM.

I give you a small list which you need to read before continuing. You can start on Wikipedia and elastic documentation.

  • syslog, the Definition of the Format and Protocol.
  • using stdout of filebeat or Logstash
  • elasticsearch, Mapping and ECS
  • kibana, benefits of having fields

Best thing would be to play around and see what happens. Every mistake helps you to understand more.

You will see a lot of Tutorials where they use grok pattern and very long Logstash config files. And the modules have a lot of common used configuration predefined. So you can use it out of the box.

Sorry but I think your question was answered. Try and get some experience.

Greetings
Malte

13

OK Malte, I'll do what you say.

Thanks and greetings !!!

14

Hello @robertitox thank you for your use of Elastic SIEM.

I am not sure if you or your team has seen this, but there is a free on-demand training course called "Elastic SIEM Fundamentals" available here:

https://www.elastic.co/training/elastic-siem-fundamentals

Please feel free to use this as an additional resource.

Thanks again!

15

Dear Mike, what you say is great, we don't know about this course.

We will try it right now, and I after that I hope I can learn in depth.

Regards!!!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.