Filebeat indices for each type of source log

Elastic Stack Beats
1

Dear all, I have an ELK 7.8.1 server with syslog and cisco module enabled from Filebeat.

At the moment I'm receiving data from:

  • Linux servers (syslog)
  • Cisco ASA (cisco module, asa data set)
  • Cisco IOS (cisco module, ios data set)

All of these data point to filebeat-* index pattern, and filebeat is the default index with 50 GB of capacity.

Please, I have two questions:

  1. Should I have to create a new index to store data coming from Linux servers, another for Cisco ASA and the last one for Cisco IOS ??? Or it's the same to point all of them to filebeat-* indices ???

  2. In case I create new indexes defining index templates, do I have to define every field name and type from each source log??? I think this is hard to do taking into account that Cisco ASA, Cisco IOS and Linux servers have several type of logs.

image

Special thanks!!!

2

Generally, you do not need to create separate indices to store different types of data being ingested by Filebeat.

The only reason you might want to do this is if you want different types of data to have different lifecycle policies. Otherwise, by default all data being ingested from Filebeat will be indexed into filebeat-* indices. The indices are named filebeat-000001, filebeat-000002, etc. Initially the data will get indexed into filebeat-000001. Once this index reaches either 50GB in size or is 30 days old, a new index named filebeat-000002 will automatically be created and data will start getting indexed there. And so on.

3

Dear Shaunak, thank you so much for your response.

Let me ask you a last question:

Having different indices for each type of data, is it better to search or store tasks performance times? Or it's the same as having just one index filebeat-* ?

Regards!!!

4

As I mentioned in my previous comment, it depends on whether you need each type of data to have different lifecycles. The other reason you might want different indices is if you want each index to have different number of shards or different number of replicas, or really any other index-level settings.

Personally I would suggest starting out with the defaults provided by Filebeat and Elasticsearch. If you start seeing specific problems or find yourself having specific use cases that aren't being satisfied with the defaults, then you can tune your setup accordingly, depending on what the specific needs are.

Shaunak

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.