Filebeat and logstash-shipper

Elastic Stack Logstash
1

Hi!

I try to integrate some new stages to our older ELK-Stack. For the new stages I would like to use filebeat and I want to connect to the logstash-shipper. But I get the error message from filebeat

18/01/29 15:35:13.061640 client.go:194: DBG  handle error: read tcp <filebeat-ip>:52566-><logstash-shipper-host-ip>:5644: i/o timeout
2018/01/29 15:35:13.061707 sync.go:78: DBG  0 events out of 2034 events sent to logstash. Continue sending
2018/01/29 15:35:13.061720 sync.go:58: DBG  close connection
2018/01/29 15:35:13.061730 client.go:110: DBG  closing
2018/01/29 15:35:13.061838 sync.go:85: ERR Failed to publish events caused by: read tcp <filebeat-ip>:52566-><logstash-shipper-host-ip>:5644: i/o timeout
2018/01/29 15:35:13.061859 single.go:91: INFO Error publishing events (retrying): read tcp <filebeat-ip>:52566-><logstash-shipper-host-ip>:5644: i/o timeout
2018/01/29 15:35:13.061870 sync.go:58: DBG  close connection
2018/01/29 15:35:13.061879 single.go:156: DBG  send fail
2018/01/29 15:35:14.062132 sync.go:53: DBG  connect
2018/01/29 15:35:14.137294 sync.go:107: DBG  Try to publish 2034 events to logstash with window size 5
2018/01/29 15:35:14.140788 client.go:194: DBG  handle error: EOF
2018/01/29 15:35:14.140818 client.go:110: DBG  closing
2018/01/29 15:35:14.140911 sync.go:78: DBG  0 events out of 2034 events sent to logstash. Continue sending
2018/01/29 15:35:14.140926 sync.go:58: DBG  close connection
2018/01/29 15:35:14.140938 sync.go:85: ERR Failed to publish events caused by: EOF
2018/01/29 15:35:14.140949 single.go:91: INFO Error publishing events (retrying): EOF
2018/01/29 15:35:14.140958 sync.go:58: DBG  close connection
2018/01/29 15:35:14.140966 single.go:156: DBG  send fail

My Filebeat.yml:

filebeat.prospectors:
- input_type: log
  paths:
      - /vol1/app1-prod1_i01_4001/log/*.log
      - /vol1/app1-prod1_i02_4002/log/*.log
   fields:
     log_type: "app1log"
     environment: "prod"
   exclude_files: [".gz$"]

 - input_type: log
   paths:
        - /vol1/app2-prod1_i01_6001/log/*.log
        - /vol1/app2-prod1_i02_6002/log/*.log
  symlinks: true
  fields:
       log_type: "2falog"
       environment: "prod"
   exclude_files: [".gz$"]
output.logstash:
    hosts: ["logstash-shipper-host:5644"]
    ssl:
               certificate_authorities: ["/etc/filebeat/logstash-forwarder.crt"]
    timeout: 30

My logstash-shipper.conf:

input {
    lumberjack {
        port => 5644
        type => "ljack"
        ssl_certificate => "/etc/pki/logstash/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/logstash/private/logstash-forwarder.key"
    }
}
output {
    redis {
    host => "localhost"
    port => 6379
    batch => true
    batch_events => 1024
    batch_timeout => 10
    congestion_interval => 1
    congestion_threshold => 20000000
    data_type => "list"
    db => 0
    key => "logstash"
  }

}

Is it even possible connect filebeat with logstash-shipper? If is it, what is my problem?

Greeting

Georg

2

Use a beats input on the Logstash side instead of an a lumberjack input.

3

That is not possible. I must use lumberjack. I can only work on the client side

4

Well, Filebeat isn't compatible with the lumberjack input.

5

thank you for your replay!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.