Best practice for shipping logs from large amount of clients using logstash

Hi guys,

I am looking for best practices for following use case:

I have a lot of clients (linux vms) which going to ship logs (and maybe metrics) over logstash to elasticseach cluster. My idea is to create one index per client. In logstash I have now possibility to create one pipeline per each client, that'd mean all clients send to dedicated port and on logstash side I just forward the logs to elasticsearch and define the respective index name. Easy. But at the end I will end up having insane amount of config files and open ports, so I will need somehow to keep the overview and maybe I will run into problems when I am using loadbalancer infront of logstash.

OR

I setup one single pipeline for all vms. All vms send logs to same port and in the pipeline I will somehow seperate the incoming loglines to the respective elasticsearch index, which means it comsumes compute power, makes ingesting slower and I end up having large config file with ifs and elses etc.

OR

I give up the idea of seperation the clients to dedicated indeces and I write all vm logs to one single index, which mean I have no opportunity to grant permissions based on the vm anymore since everything is in one index and when I want to delete logs from a specific machine I would somehow need to delete documents from one big index, which is expensive

Does anyone of you have some advice how this can be solved?

I can cre

It depends on which input you are using in logstash, but if it is an input that defines the client name as a field on the event it would be trivial to use that as the index name for the elasticsearch output.

input is always "beats" for all sources

If your beat is adding host metadata then should have a [host][[name] field on the event that you can use as the index name.

1 Like