Filebeat doesn't send logs to logstash in client server architecture

Elastic Stack Beats
1

Hi all,

I just set up filebeat following the documentation. But I can't have logs of my cluster in logstash. I use a client-server architecture (i.e a client with Filebeat configured and installed and a server with logstash and elasticsearch) The client and the server are not in the same subnet so I used domain name to reach logstash in the Server instance.

Here is my /etc/filebeat/filebeat.yml config file:

`filebeat:
  prospectors:
    -
      paths:
        - "/var/log/*.log"

    input_type: log

output:
  logstash:
    hosts: ["filebeatdomainname.com:5044"]
    index: indexname

shipper:

logging:
  files:
  rotateeverybytes: 10485760 # = 10MB`

And on logstash side, I just configured a logstash.conf file under /etc/logstash/conf.d/logstash.conf:

`input {
 beats {
   port => 5044
}
}

output {
elasticsearch {
hosts => "172.31.62.244:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}`

Please, Were I am wrong?

Thanks in advance!

2

Any log output from filebeat/logstash?

3

Yes, Any log.

4

I means no logs in logstash and no index in elasticsearch

5

can you post filebeat log output?

6

Check out filebeat troubleshooting/faq: https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#filebeat-not-collecting-lines

7

I don't know where are filebeat log output. I expected it to be on /var/log/filebeat but I don't have it here. Please, do you know where I can see them?

8

Hey @steffens, Here is what I have in the logs

2016/03/20 17:26:23.768081 output.go:87: DBG output worker: publish 177 events 2016/03/20 17:26:23.768272 client.go:136: DBG Try to publish 177 events to logstash with window size 10 2016/03/20 17:26:23.890194 client.go:95: DBG close connection 2016/03/20 17:26:23.890362 client.go:114: DBG 0 events out of 177 events sent to logstash. Continue sending ... 2016/03/20 17:26:23.890398 single.go:76: INFO Error publishing events (retrying): EOF 2016/03/20 17:26:23.890423 single.go:152: INFO send fail 2016/03/20 17:26:23.890447 single.go:159: INFO backoff retry: 1s 2016/03/20 17:26:24.241759 reader.go:138: DBG End of file reached: /var/log/auth.log; Backoff now. 2016/03/20 17:26:24.241951 reader.go:138: DBG End of file reached: /var/log/cloud-init-output.log; Backoff now. 2016/03/20 17:26:24.242127 reader.go:138: DBG End of file reached: /var/log/boot.log; Backoff now. 2016/03/20 17:26:24.242734 reader.go:138: DBG End of file reached: /var/log/dpkg.log; Backoff now. 2016/03/20 17:26:24.242906 reader.go:138: DBG End of file reached: /var/log/cloud-init.log; Backoff now. 2016/03/20 17:26:24.243992 reader.go:138: DBG End of file reached: /var/log/kern.log; Backoff now. 2016/03/20 17:26:24.890672 client.go:90: DBG connect 2016/03/20 17:26:25.012786 client.go:136: DBG Try to publish 177 events to logstash with window size 5 2016/03/20 17:26:25.136129 client.go:95: DBG close connection 2016/03/20 17:26:25.136279 client.go:114: DBG 0 events out of 177 events sent to logstash. Continue sending ... 2016/03/20 17:26:25.136319 single.go:76: INFO Error publishing events (retrying): EOF 2016/03/20 17:26:25.136345 single.go:152: INFO send fail

9

Issue fixed. Thank you for your tip. After reading deeply the logs I saw that Filebeat was not able to communicate directly with logstash. So in logstash logs I saw a warning. I updated the beats plugin and everything works fine. Thanks again!!!!

10

yo dawg. same problem, that link is rotten now though. what was the secret sauce?

1 Like
11

Here's the updated link: https://www.elastic.co/guide/en/beats/filebeat/current/faq.html

1 Like