Best practice for shipping logs from large amount of clients using logstash

Elastic Stack Logstash
1

Hi guys,

I am looking for best practices for following use case:

I have a lot of clients (linux vms) which going to ship logs (and maybe metrics) over logstash to elasticseach cluster. My idea is to create one index per client. In logstash I have now possibility to create one pipeline per each client, that'd mean all clients send to dedicated port and on logstash side I just forward the logs to elasticsearch and define the respective index name. Easy. But at the end I will end up having insane amount of config files and open ports, so I will need somehow to keep the overview and maybe I will run into problems when I am using loadbalancer infront of logstash.

OR

I setup one single pipeline for all vms. All vms send logs to same port and in the pipeline I will somehow seperate the incoming loglines to the respective elasticsearch index, which means it comsumes compute power, makes ingesting slower and I end up having large config file with ifs and elses etc.

OR

I give up the idea of seperation the clients to dedicated indeces and I write all vm logs to one single index, which mean I have no opportunity to grant permissions based on the vm anymore since everything is in one index and when I want to delete logs from a specific machine I would somehow need to delete documents from one big index, which is expensive

Does anyone of you have some advice how this can be solved?

I can cre

2

It depends on which input you are using in logstash, but if it is an input that defines the client name as a field on the event it would be trivial to use that as the index name for the elasticsearch output.

3

input is always "beats" for all sources

4

If your beat is adding host metadata then should have a [host][[name] field on the event that you can use as the index name.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.