Logstash ignore the port defined in pipeline


(Raphael Amaral) #1

Dear,

I would like receive help to understand a behavior...

My infrastructure is Filebeat -> Logstash -> Elasticsearch -> Kibana

Filebeat:
We have many server send documents to Logstash using differents pipelines and specific ports.

Logstash:
We have 5 pipelines with differents ports.

In Logstash we need to use a IF by [TYPE] in Filter and Output blocks because without these IF all pipelines receive all documents sent by filebeat even using differents ports.

Logstash version is 5.1.2.

Exemple for Logstash and Filebeat configuration:

Logstash:
input {
beats {
port => "5005"
}
}

filter {
if [type] == "log_prod" {
json {
source => "message"
remove_field => [ "host", "beat", "message", "tags", "source", "input_type" ]
}
}

date {
timezone => "America/Sao_Paulo"
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
if [type] == "log_prod" {
elasticsearch {
hosts => ["server1:9200", "server2:9200"]
index=> "index-%{+YYYY.MM.dd}"
}
}
}

Filebeat:

filebeat.prospectors:

  • input_type: log
    paths:
  • /var/log/kibana_log/kibana.log
    document_type: log_prod
    scan_frequency: 30s
    output:
    logstash:
    hosts: ["server_logstash:5005"]
    logging:
    to_syslog: false
    to_files: true
    files:
    path: /var/log/filebeat
    name: filebeat.log
    keepfiles: 5
    level: info

(Magnus B├Ąck) #3

What's your question?


(Raphael Amaral) #4

Is normal this behavior ignoring the port defined in input block?


(Leandro Maciel) #5

if i understood your question, It is normal behavior.

Logstash only has one pipeline, you can have multiple .conf files in the conf.d directory, but when logstash is started it will concatenated all the files in one file.

For example, if you have two servers running filebeat, each of one sending to a different port, like 5001 and 5002, you will need to start two beats listeners in logstash.

input {
    beats {
        port => "5001"
    }
    beats {
        port => "5002"
    }
}

It will work, but if you need to send the different inputs to different indices, you will need to filter by type in the filter block or in the output block.

Without conditional statements in the filter and output blocks Logstash will not differentiate the inputs based on its source.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.