Logstash ignore the port defined in pipeline

[camarar (Raphael Amaral)]

Dear,

I would like receive help to understand a behavior...

My infrastructure is Filebeat -> Logstash -> Elasticsearch -> Kibana

Filebeat:
We have many server send documents to Logstash using differents pipelines and specific ports.

Logstash:
We have 5 pipelines with differents ports.

In Logstash we need to use a IF by [TYPE] in Filter and Output blocks because without these IF all pipelines receive all documents sent by filebeat even using differents ports.

Logstash version is 5.1.2.

Exemple for Logstash and Filebeat configuration:

Logstash:
input {
beats {
port => "5005"
}
}

filter {
if [type] == "log_prod" {
json {
source => "message"
remove_field => [ "host", "beat", "message", "tags", "source", "input_type" ]
}
}

date {
timezone => "America/Sao_Paulo"
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
if [type] == "log_prod" {
elasticsearch {
hosts => ["server1:9200", "server2:9200"]
index=> "index-%{+YYYY.MM.dd}"
}
}
}

Filebeat:

filebeat.prospectors:

• input_type: log
  paths:
• /var/log/kibana_log/kibana.log
  document_type: log_prod
  scan_frequency: 30s
  output:
  logstash:
  hosts: ["server_logstash:5005"]
  logging:
  to_syslog: false
  to_files: true
  files:
  path: /var/log/filebeat
  name: filebeat.log
  keepfiles: 5
  level: info

[magnusbaeck (Magnus Bäck)]

What's your question?

[camarar (Raphael Amaral)]

Is normal this behavior ignoring the port defined in input block?

[leandrojmp (Leandro Pereira)]

if i understood your question, It is normal behavior.

Logstash only has one pipeline, you can have multiple .conf files in the conf.d directory, but when logstash is started it will concatenated all the files in one file.

For example, if you have two servers running filebeat, each of one sending to a different port, like 5001 and 5002, you will need to start two beats listeners in logstash.

input {
    beats {
        port => "5001"
    }
    beats {
        port => "5002"
    }
}

It will work, but if you need to send the different inputs to different indices, you will need to filter by type in the filter block or in the output block.

Without conditional statements in the filter and output blocks Logstash will not differentiate the inputs based on its source.