# Best practice for shipping logs from large amount of clients using logstash

**URL:** <https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207>\
**Category:** Logstash\
**Created:** [July 30, 2020, 11:09am UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207 "2020-07-30T11:09:52Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\
**Post date:** [July 30, 2020, 11:09am UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207/1 "2020-07-30T11:09:52Z")

</div>

Hi guys,

I am looking for best practices for following use case:

I have a lot of clients (linux vms) which going to ship logs (and maybe metrics) over logstash to elasticseach cluster. My idea is to create one index per client. In logstash I have now possibility to create one pipeline per each client, that'd mean all clients send to **dedicated port** and on logstash side I just forward the logs to elasticsearch and define the respective index name. Easy. But at the end I will end up having insane amount of config files and open ports, so I will need somehow to keep the overview and maybe I will run into problems when I am using loadbalancer infront of logstash.

OR

I setup one single pipeline for all vms. All vms send logs to **same port** and in the pipeline I will somehow seperate the incoming loglines to the respective elasticsearch index, which means it comsumes compute power, makes ingesting slower and I end up having large config file with ifs and elses etc.

OR

I give up the idea of seperation the clients to dedicated indeces and I write all vm logs **to one single index** , which mean I have no opportunity to grant permissions based on the vm anymore since everything is in one index and when I want to delete logs from a specific machine I would somehow need to delete documents from one big index, which is expensive

Does anyone of you have some advice how this can be solved?

I can cre

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 30, 2020, 5:27pm UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207/2 "2020-07-30T17:27:51Z")

</div>

> [@Kosodrom](#):
>
> All vms send logs to **same port** and in the pipeline I will somehow seperate the incoming loglines to the respective elasticsearch index

It depends on which input you are using in logstash, but if it is an input that defines the client name as a field on the event it would be trivial to use that as the index name for the elasticsearch output.

---

<div class="post-metadata">

**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\
**Post date:** [July 31, 2020, 7:47am UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207/3 "2020-07-31T07:47:57Z")

</div>

input is always "beats" for all sources

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2020, 2:24pm UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207/4 "2020-07-31T14:24:37Z")

</div>

If your beat is adding host metadata then should have a [host][[name] field on the event that you can use as the index name.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 28, 2020, 2:24pm UTC](https://discuss.elastic.co/t/best-practice-for-shipping-logs-from-large-amount-of-clients-using-logstash/243207/5 "2020-08-28T14:24:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.