Logstash configuration best practices

[Dzious]

Hi everyone,

I would like to know a bit more about logstash best practices
I need to setup a monitoring system for about 20 servers at the moment and more to come in the future.

Onto these servers, i'll have multiple things i would like to get as logstash input : Metrics, Logs and Dadabase. I do not need to get all of them on all servers but i'll mainly have at least 2 out of the 3 modules enable.

Here is my question. Should i make a file for each server with all inputs, filters and outputs or is it better to have a single file for each kind of metric + a file for filter + a file for output ?

If none of the idea is good, what are your advice.

Thanks by advance

[A_B]

Hi @Dzious ,

you probably want something like Filebeat (or winlogbeat) to collect the logs. Many services have Filebeat modules that will help with parsing logs. That would mean less filtering in Logstash (possibly).

For metrics there is also Metricbeat.

If you use Filebeat for logs, you probably only need one Logstash beats input. Do you use any sort of config management system, like Puppet, Ansible or Salt?

[Dzious]

Thanks for your reply

I actually use Metricbeat and Filebeat is planned to be add so my beat input is good at the moment
But since i will mainly ingest logs from in house apps that do not have a module i will need to have a lot of filters anyway. (and i can't use ECS compatible loggers. Impementing it in my dev environnement and going through all the steps to the dev would need to much work and time at the moment)

Also my problem is mainly cause by database input that will need at least one jdbc configuration per database.

Do you use any sort of config management system, like Puppet, Ansible or Salt?

I do not use any of these. I didn't even know about it

[A_B]

@Dzious

Here is my question. Should i make a file for each server with all inputs, filters and outputs or is it better to have a single file for each kind of metric + a file for filter + a file for output ?

Here is my personal experience. Not sure how well it lines up with possible best practices.

We used to use one inputs config file, one filters config file and one outputs config file and the same files were used in all of our environments (development, staging and production). Over time they became very complex, especially the filters config. It was also unnecessarily difficult to make environment specific overrides or to test new things in the development environment without possibly breaking the production environment. This is partly because of how we have setup out configuration management system...

Personally I like our current implementation much better than what I just described. We split up the config so that each input, filter and output have their own config file, which makes them like lego blocks. We also use Logstash pipelines so that we can choose which messages go through which filters without always having to use some sort of conditional.

In our configuration management system I can make a list of pipelines and which Logstash configurations they should have, which could look like this

pipeline1:
- input_beats
- filter_drop_fields
- filter_grok_nginx
- output_elasticsearch
pipeline2:
- input_udp
- filter_fix_date
- filter_drop_fields
- output_elasticsearch

I hope that makes sense... This has been a slow development over many years and would need some sort of automates config deployment.

Note, that if you attempt something like this, the Logstash configuration file for each pipeline is concatenated at startup as if you would do cat * in the folder where the config fragments are, so file naming will be important to get the correct order of especially filters if you require a certain order or chain of filters.

And now that I reread the OP, maybe this is a bit overboard for 20 servers...

Anyway, this is where I have ended up after 4-5 years with Logstash