Fails to receive any log events，when two piplines using the same input port 5044

Elastic Stack Logstash
1

I encountered a troubling issue for which I'd like to express my gratitude to anyone who can help. One pipeline consistently failed to receive log events. The problem occurred when two pipelines used the same input port, e.g. 5044. Additionally, the Beats for these two pipelines were coming from different servers.

Here is the configuration for the pipelines:


- pipeline.id: bi
  path.config: "{{logstash_pipeline_conf_dir}}/prod-bi-es.conf"

- pipeline.id: easy-bi
  path.config: "{{logstash_pipeline_conf_dir}}/prod-easy-bi-es.conf"

The configuration for prod-bi-es.conf is as follows:

input {
  beats {
    id => "prod-bi-log-1c"
    port => 5044
    codec => "plain"
    client_inactivity_timeout => 36000
  }
}
. . .
output {
    if ([fields][sys-name] == "bi" and [fields][env] == 'prod') {
        elasticsearch {
. . .

The configuration for prod-easy-bi-es.conf is as follows:

input {
  beats {
    id => "prod-easy-bi-log-1c"
    port => 5044
    codec => "plain"
    client_inactivity_timeout => 36000
  }
}
. . .
output {
    if ([fields][sys-name] == "easy-bi" and [fields][env] == 'prod') {
        elasticsearch {
. . .

After starting Logstash, one of the pipelines fails to receive any events, and the log data isn't being stored in Elasticsearch using the output plugin.

However, when I modified the input port for one of the profiles, for instance, changing it to 5045 ,and, I've also changed the corresponding Filebeat ports, I was able to successfully receive data from the Beats.

If you've ever encountered a similar problem, please help me navigate through it, thanks in advance!

Extended information (the version of the software I'm using)：

  • Filebeat 7.10.0
  • Logstash 7.10.0
  • Elasticsearch 6.8
2

This is not possible, you can't use the same port for different inputs.

3

As Leandro said, 2 input beats on the same port, impossible.
You can:

  • set the single port, and based on a message content, fields from FB - [fields][sys-name] or make additional fields or tags
  • receive on the single port, forward to different pipelines based on [fields][sys-name] from FB or the content

PS. Elasticsearch 6.8 is deprecated.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.