# Logstash Multi pipeline with beats input

**URL:** <https://discuss.elastic.co/t/logstash-multi-pipeline-with-beats-input/308612>\
**Category:** Logstash\
**Tags:** ingest-pipeline\
**Created:** [June 30, 2022, 8:30pm UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-with-beats-input/308612 "2022-06-30T20:30:05Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![MM2022](https://avatars.discourse-cdn.com/v4/letter/m/d26b3c/32.png) [@MM2022](https://discuss.elastic.co/u/MM2022)\
**Post date:** [June 30, 2022, 8:30pm UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-with-beats-input/308612/1 "2022-06-30T20:30:05Z")

</div>

Hi All,

I have one ELK stack and multiple clients to send their data to it. Each client should have their own Index and clients should not be able to see each others data. I am using filebeats on the client servers. the config is pretty simple , in the filebeat input (on the client side) I use  
fileds:  
source: "client1server"

* * *

in Logstash pipelines.yml:

- pipeline.id: beats  
config.string: |  
input {  
beats {  
port =\> 5044  
ssl =\> true  
ssl\_certificate\_authorities =\>["/etc/logstash/config/certs/elasticsearch-ca.pem"]  
ssl\_key =\> '/etc/logstash/config/certs/logstash-pkcs8.key'  
ssl\_certificate =\> '/etc/logstash/config/certs/logstash.crt'  
}  
}  
output {  
if [source] == 'client1server' {  
pipeline { send\_to =\> client1 }  
} else if [source] == 'client2server' {  
pipeline { send\_to =\> client2 }  
}  
}
- pipeline.id: client1  
path.config: "/etc/logstash/conf.d/client1.conf"
- pipeline.id: client2  
path.config: "/etc/logstash/conf.d/client2.conf"

* * *

## and the clientx.conf looks like this: input { pipeline { address =\> client1 } } output { elasticsearch { hosts =\> ["[https://x:9200](https://x:9200)","[https://x:9200](https://x:9200)","x:9200"] index =\> "client1-%{+YYYY.MM.dd}" cacert =\> "/etc/logstash/config/certs/elasticsearch-ca.pem" ssl\_certificate\_verification =\> true user =\> 'user' password =\> "${elasticsearch.password}" } }

I manage to bring logstash fine (although I have to bring it up from cli for some reason if I just do systemctl start logstash it will be up but not listening to 5044).  
After it is listening to 5044 I start the filebeat on my client side it does create 2 indices (one for client1 and one for client2) but I see the data from both servers in both indices!!!! As if the condition in the output of piplines.yml is being ignored ! I see the fields.source with both server names in both data views in Kibana!  
what am I missing? why the condition is not working properly? I'd appreciate any help.

---

<div class="post-metadata">

**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\
**Post date:** [July 1, 2022, 1:45am UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-with-beats-input/308612/2 "2022-07-01T01:45:03Z")

</div>

I never used pipeline.yml like that.  
this is how I use it.

pipeline.yml is simple two liner, config file and name of pipeline id.

then in config file

```auto
input {
   beats {
        port => 5044
        ssl => true
        ssl_certificate_authorities =>["/etc/logstash/config/certs/elasticsearch-ca.pem"]
        ssl_key => '/etc/logstash/config/certs/logstash-pkcs8.key'
        ssl_certificate => '/etc/logstash/config/certs/logstash.crt'
     }
}

```

in filter section I do something like this

```auto
filter {
  if source is from client1 
           mutate { add_field => { "[@metadata][target_index]" => "client1-indexname" } }
  if source is from client2
           mutate { add_field => { "[@metadata][target_index]" => "client2-indexname" } }
}

## output {
elasticsearch {
   hosts => ["[https://x:9200](https://x:9200/)","[https://x:9200](https://x:9200/)","x:9200"]
   index => "%{[@metadata][target_index]}"
   cacert => "/etc/logstash/config/certs/elasticsearch-ca.pem"
   ssl_certificate_verification => true
   user => 'user'
   password => "${elasticsearch.password}"
  }
}

```

and logstash pipeline will run as one. takes all the data from all client. but sends them to proper index

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 29, 2022, 1:45am UTC](https://discuss.elastic.co/t/logstash-multi-pipeline-with-beats-input/308612/3 "2022-07-29T01:45:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.