Asking for Filebeat and Logstash configuration design advice

Elastic Stack Elasticsearch
1

Hello,
I am new to Elastic stack so please be gentle :slight_smile:
I'd like to ask you for an advice for Filebeat and Logstash configuration in the following world:
There are X servers: S1, S2, S3, ... Sx
Servers have the following logs to be shipped to Elastic stack:

  • system logs (/var/log/*)
    and combination of application logs:
  • app1 logs (/opt/app1/logs/*)
  • app2 logs (/opt/app2/logs/*)
  • ...
  • appX logs (/opt/appX/logs/*)
    My question is how to organize filebeat and logstash configuration to be able in Kibana/Elasticsearch:
  • quickly find logs from appX from Sx server
  • quickly find logs from appX across Sx - Sy servers
  • quickly find logs from system logs from Sx server
  • quickly find logs from system logs from Sx - Sy server

How about indices ? Should I put system logs from each server into different index (X indices) or one index combined for all servers ? The same for applications: one index for all appX across all servers or one index for appX for each server ?

2

you might want to ask this in the respective beats and logstash forum, as the chance us much higher to get help, compared to Elasticsearch forum.

I also refrained from moving this post, as it kind of belongs in both groups. So please go ahead yourself.

Thanks!

3

You can add a field to each prospector indicating the application already when you collect data in Filebeat. If you combine this with a field that contains a server identifier at the instance level you can use these two parameters in Kibana to filter by server and/or application, Sio there is no need to have separate indices.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.