Logstash can't handle multiple pipline

[Robin_Guo]

Specs:

logstash  Version:5.5.1
OS:centos 7.3.1611-x86_64

logstash pipline conf

@/etc/logstash/conf.d/metricbeat.conf

#logstash for metricbeat
input {
  beats {
    port => 5044
  }
}

filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{agent}-%{env}-%{dc}-%{os}-%{service}" } }
   mutate  {  remove_field =>  ["agent","dc","env","os","service","type"]  }

}
 

output {
    file {
    path => "/tmp/logstash-5044"
    }
}

@/etc/logstash/conf.d/tcpbeat.conf

#logstash for tcpbeat
input {
  tcp {
    port => 5050
    codec => "json"
  }
}

filter {
  json {
    source => "message"
  }
  mutate {
         remove_field =>  ["message"]  
  }
}


output {
     file {
         path =>"/tmp/logstash-5050"
     }
}

test case

#json file

@/tmp/test
{"agent":"metricbeat","env":"production","dc":"fr4","os":"Linux","service":"pgpool","beat":{"name":"robinguo-HP-Z210-Workstation"},"pgpool_active":20,"pgpool_idle":80,"pgpool_waiting":100}

#test
nc fr4eslogstash01.tls.ad 5050 < /tmp/test

#results
@/tmp/logstash-5044
{"pgpool_active":20,"pgpool_waiting":100,"@timestamp":"2017-08-18T06:40:32.847Z","port":38428,"beat":{"name":"robinguo-HP-Z210-Workstation"},"pgpool_idle":80,"@version":"1","host":"10.65.186.63"}

@/tmp/logstash-5050
{"pgpool_active":20,"pgpool_waiting":100,"@timestamp":"2017-08-18T06:40:32.847Z","port":38428,"beat":{"name":"robinguo-HP-Z210-Workstation"},"pgpool_idle":80,"@version":"1","host":"10.65.186.63"}

Problem:

Why logstash pipline can't distinguish from different tcp request, I just sent a json test file to logstash tcp 5050,why
tcp 5044 also received the message. it's not what I wanted results. if I enabled 10 or 12 piplines with different port sepereatly, when I just sent some data to a particular port of logstash, all other ports also can got the data, that's so mess up.

Does anyone can give a solution to address this issue?

best regards

Robin

[magnusbaeck (Magnus Bäck)]

Unless you have a very recent Logstash and explicitly configure its recently added multi-pipeline feature you only have a single pipeline. To choose which filters and outputs to apply to which inputs you need to use conditionals.

[Robin_Guo]

hi @magnusbaeck,
cuz 6.0 is still in beta version, all of ＥＬＫ we currently are using v5.5.1, so it's not easy to upgrade.. I would like to keep current version and using conditional filters to implement it or enable multiple logstash instances.
Would you please give me a detailed example walk me through that?

Thanks

[magnusbaeck (Magnus Bäck)]

This question comes up every week and I don't have time to answer in detail every time. Please search the archives of this forum and make sure you understand https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html.