Filebeat sends logs, but only at the second try

Elastic Stack Beats
1

Hello!

So. It seems our case's similar to that (Filebeat EOF - And then succeeds at second try). We have errors in filebeat stating that:

2016-12-27T18:25:51+03:00 DBG  Try to publish 2 events to logstash with window size 1
2016-12-27T18:25:51+03:00 DBG  handle error: EOF
2016-12-27T18:25:51+03:00 DBG  closing
2016-12-27T18:25:51+03:00 DBG  0 events out of 2 events sent to logstash. Continue sending
2016-12-27T18:25:51+03:00 DBG  close connection
2016-12-27T18:25:51+03:00 ERR Failed to publish events caused by: EOF
2016-12-27T18:25:51+03:00 INFO Error publishing events (retrying): EOF
2016-12-27T18:25:51+03:00 DBG  close connection
2016-12-27T18:25:51+03:00 DBG  send fail
2016-12-27T18:25:52+03:00 DBG  connect
2016-12-27T18:25:52+03:00 DBG  Try to publish 2 events to logstash with window size 1
2016-12-27T18:25:52+03:00 DBG  1 events out of 2 events sent to logstash. Continue sending
2016-12-27T18:25:52+03:00 DBG  Try to publish 1 events to logstash with window size 1
2016-12-27T18:25:52+03:00 DBG  1 events out of 1 events sent to logstash. Continue sending
2016-12-27T18:25:52+03:00 DBG  send completed
2016-12-27T18:25:52+03:00 DBG  Events sent: 2
2016-12-27T18:25:52+03:00 DBG  Processing 2 events
2016-12-27T18:25:52+03:00 DBG  Registrar states cleaned up. Before: 20, After: 20
2016-12-27T18:25:52+03:00 DBG  Write registry file: /var/lib/filebeat/registry

In other words logs'are sent, but after failing once. And there's nothing in Logstash logs.

Filebeat config (actually there're many more prospectors, but the differences only in the path and quantity of the prospectors):

############################# Filebeat ######################################
filebeat:
  prospectors:
    -
      paths:
        - /var/log/problem*.log
      input_type: log
      document_type: problem_log
    -
      paths:
        - /var/log/error-*.log
      input_type: log
      document_type: error_log
      multiline:
        pattern: '^[0-9]{6}_[0-9]{6}'
        negate: true
        match: after
      exclude_lines: [ '^\n' ]

  registry_file: /var/lib/filebeat/registry

############################# Output ##########################################

output:
  logstash:
    protocol: https
    hosts: ["IP:5044"]
    worker: 1
    compression_level: 3
    loadbalance: true
    ssl:
      certificate_authorities: ["/etc/filebeat/logstash.crt"]
############################# Logging #########################################
logging:
  files:
    # The directory where the log files will written to.
    #path: /var/log/mybeat

    # The name of the files where the logs are written to.
    #name: mybeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

  # Sets log level. The default log level is error.
  # Available log levels are: critical, error, warning, info, debug
  #level: error

What could be the cause? Maybe we should add/edit some parameters? Could you give us some tips, please?

2

The fail is due to EOF (end of file). That is either logstash itself or some other device/software did close the connection between beats and logstash. Maybe because logstash did take too much time.

Which logstash, filebeat version are you using? If you're using LS 2.4 or 5.x, update the logstas-input-beats plugin to the most recent version.

3

What's EOF? How does it correlate with the connection? Maybe we should set some timeout?

The whole ELK stack consist of 5.x versions.

4

EOF is short for End Of File. This means, no more content can be read/written as the underlying file descriptor has been close by the OS.

Which versions are you using exactly? Have you update the logstash-input-beats plugin? Which exact version of logstash-input-beats have you installed?

5

Hello!

Logstash: 5.0.0
Filebeat: 5.1.1

Regarding EOF. It doesn't imply the actual end of a file (no more lines, no more data), but just the command from the system? So the file not necessarily can be read completely and some data may remain? Am I understanding that correctly?

6

The EOF you see is related to the connection to Logstash and the file you are reading itself. So you should investigate what goes wrong with the connection to LS. The version of the logstash-beats-plugin is also important here.

Check if you see some errors / logs on the logstash side that could help here. Also there are quite a few existing topics with similar issues which were solved: https://discuss.elastic.co/search?q=Failed%20to%20publish%20events%20caused%20by%3A%20EOF Perhaps one of these can help you solve your problem.

7

Logstash 5.0.0 does not ship with most recent logstash-input-beats plugin.

From plugin changelog it seems you want to have at least 3.1.11 installed. Use ./bin/logstash-plugin to update the input plugin.

8

Oh, I didn't know about plugin updating. Ok, I'll give a try then. Thanks!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.