Config files for mutliple pipelines

Elastic Stack Logstash
1

Hi,
I am trying to configure two separate pipelines in logstash-6.1.1 through pipeline.yml. For each configuration file i am specifying same input and output.

1st_pipeline.conf:-
input{
beats{
port => 5044
}
}

filter{
....................
}

output{
stdout{ codec => rubydebug}
}

2nd_pipeline.conf:-
input{
beats{
port => 5044
}
}

filter{
.................................
}

output{
stdout{ codec => rubydebug }
}

Now as i am trying to run both of these pipeline together its throwing error

Plugin: <LogStash::Inputs::Beats port=>5044, id=>"289fbe7dcd840f0b3bf8e6a2cc6ef3f98776d60fcb8cc214b41d85f129b283a3", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_2796f793-4d5f-4c9c-a464-86c091b70277", enable_metric=>true, charset=>"UTF-8">, host=>"0.0.0.0", ssl=>false, ssl_verify_mode=>"none", include_codec_tag=>true, ssl_handshake_timeout=>10000, tls_min_version=>1, tls_max_version=>1.2, cipher_suites=>["TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384", "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256"], client_inactivity_timeout=>60, executor_threads=>16>
Error: Address already in use

2

You can not have two different inputs listening on the same port within a Logstash instance, even if they belong to different pipelines. You will therefore need to change the port for one of them.

3

i changed the the port it works properly.
thanks

4

Hi @Christian_Dahlqvist.

If we changed the port for one of pipelines what about the filebeat.yml do we need to change anything in the logstash output. for example my filebeat.yml has output.logstash as follow:
output.logstash:
# The Logstash hosts
hosts: ["logstashdomain:5044"]
and my filebeat shipping three different log events (apache error, apache acces, mysql error)

and on the logstash server I have three pipelines
and each of this conf file has beats input with port 5044. So if i give unique port to each of them do i need to edit my output.logstash in filebeat.yml for example:
output.logstash:
# The Logstash hosts
hosts: ["logstashdomain:5044", "logstashdomain:5045", "logstashdomain:5046" ]

Is it correct?

5

Hi @shubham_kumar1 can you please help with this since you had experience with this issue.

If we changed the port for one of pipelines what about the filebeat.yml do we need to change anything in the logstash output. for example my filebeat.yml has output.logstash as follow:
output.logstash:
# The Logstash hosts
hosts: ["logstashdomain:5044"]
and my filebeat shipping three different log events (apache error, apache acces, mysql error)

and on the logstash server I have three pipelines
and each of this conf file has beats input with port 5044. So if i give unique port to each of them do i need to edit my output.logstash in filebeat.yml for example:
output.logstash:
# The Logstash hosts
hosts: ["logstashdomain:5044", "logstashdomain:5045", "logstashdomain:5046" ]

Is it correct?

6

Yes you need to update the port in filebeat also. logstash will only listen to the port mentioned in input.

1 Like
7

thank you @shubham_kumar1 for your help. Actually we decided to use two filebeat instances.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.