Which version of logstash support for multiple which version of logstash can support multiple pipline?

Elastic Stack Logstash
1

Hi ES team,

I wonder know , which version of Logstash can support multiple pipline within a group of logstash server by now?

Currently we used few beats on monitored side ,whatever metricbeat/filebeat/heartbeat or the beat we build by ourselves.

But we have to assign a dedicated logstash cluster for a group of particular beat.

My question is that can newer logstash support more multiple pipline instead of one beat corresponding to one logstash?

Thanks

for example:

#/etc/logstash/conf.d

  • filebeat.conf
#logstash for filebeat
input {
  beats {
    port => 5044
  }
}
 
filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{env}-%{dc}-%{os}-%{service}" } }
   mutate  {  remove_field =>  ["dc","env","os","service","type"]  }

}


output {
  elasticsearch {
     hosts => ["estest01.tls.ad:9200","estest02.tls.ad:9200"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
     sniffing => false
     
   }
}
  • heartbeat.conf
#logstash for heartbeat
input {
  beats {
    port => 5045
  }
}
 
filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{env}-%{dc}-%{os}-%{service}" } }
   mutate  {  remove_field =>  ["dc","env","os","service","type"]  }

}


output {
  elasticsearch {
     hosts => ["estest03.tls.ad:9200","estest04.tls.ad:9200"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
     sniffing => false
     
   }
}
  • metricbeat.conf
#logstash for metricbeat
input {
  beats {
    port => 5046
  }
}
 
filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{env}-%{dc}-%{os}-%{service}" } }
   mutate  {  remove_field =>  ["dc","env","os","service","type"]  }

}


output {
  elasticsearch {
     hosts => ["estest05.tls.ad:9200","estest06.tls.ad:9200"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
     sniffing => false
     
   }
}
  • tcpbeat.conf
#logstash for tcp
input {
  tcp {
    port => 5047
  }
}
 
filter {
   mutate  {  add_field => { "[@metadata][index_prefix]" => "%{env}-%{dc}-%{os}-%{service}" } }
   mutate  {  remove_field =>  ["dc","env","os","service","type"]  }

}


output {
  elasticsearch {
     hosts => ["estest07.tls.ad:9200","estest08.tls.ad:9200"]
     template_overwrite => false
     manage_template => false
     index => "%{[@metadata][index_prefix]}-%{+YYYY.MM.dd}"
     sniffing => false
     
   }
}
2

Multi-pipeline support was added in Logstash 6.0.

But we have to assign a dedicated logstash cluster for a group of particular beat.

Why do you think you have to do that?

3

Because currently we used v5.5.2, It doesn't support multiple pipline yet.
So we have to implement multiple pipline conf with a few groups of logstash clusters. which each pipline conf with a group of logstash cluster.

We distinguish between different kind of beats through the port instead of field type/flag by far. we use same port for same beat service.

is that clear?

Do you have any good suggestions for that?

Thanks in advance

Thanks

4

We distinguish between different kind of beats through the port instead of field type/flag by far. we use same port for same beat service.

I think that's an unnecessary complication.

Do you have any good suggestions for that?

If you insist on using different ports and want to isolate different kinds of events then using the multi-pipeline feature is a good idea, but I wouldn't do it myself.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.