ERR Failed to publish events caused by: EOF

Elastic Stack Beats
1

After reinstall all components, now the logs are written into ES by logstash, but the error message still exists. Not sure if it is safe to ignore it.
----------------- EDIT -----------------------

Hi,
I am running version 5.0 of filebeat, logstash and Elasticsearch on one server. When I config filebeat output to ES, it works, but after I change output to logstash, it stops working.

The os is Ubuntu 14.04.3 LTS，logs and configs as following:

/var/log/logstash/logstash-plain.log

[2016-11-01T14:53:02,685][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2016-11-01T14:53:02,812][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2016-11-01T14:53:03,025][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>["http://192.168.199.9:9200"]}}
[2016-11-01T14:53:03,026][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["192.168.199.9:9200"]}
[2016-11-01T14:53:03,028][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2016-11-01T14:53:03,039][INFO ][logstash.pipeline ] Pipeline main started
[2016-11-01T14:53:03,067][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

/var/log/filebeat/filebeat

2016-11-01T14:56:31+08:00 INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2016-11-01T14:56:31+08:00 INFO Setup Beat: filebeat; Version: 5.0.0
2016-11-01T14:56:31+08:00 INFO Max Retries set to: 3
2016-11-01T14:56:31+08:00 INFO Activated logstash as output plugin.
2016-11-01T14:56:31+08:00 INFO Publisher name: fd-dev
2016-11-01T14:56:31+08:00 INFO Flush Interval set to: 1s
2016-11-01T14:56:31+08:00 INFO Max Bulk Size set to: 2048
2016-11-01T14:56:31+08:00 INFO filebeat start running.
2016-11-01T14:56:31+08:00 INFO Registry file set to: /var/lib/filebeat/registry
2016-11-01T14:56:31+08:00 INFO Loading registrar data from /var/lib/filebeat/registry
2016-11-01T14:56:31+08:00 INFO States Loaded from registrar: 5
2016-11-01T14:56:31+08:00 INFO Loading Prospectors: 1
2016-11-01T14:56:31+08:00 INFO Start sending events to output
2016-11-01T14:56:31+08:00 INFO Load previous states from registry into memory
2016-11-01T14:56:31+08:00 INFO Starting Registrar
2016-11-01T14:56:31+08:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016-11-01T14:56:31+08:00 INFO Previous states loaded: 5
2016-11-01T14:56:31+08:00 INFO Loading Prospectors completed. Number of prospectors: 1
2016-11-01T14:56:31+08:00 INFO All prospectors are initialised and running with 5 states to persist
2016-11-01T14:56:31+08:00 INFO Starting prospector of type: log
2016-11-01T14:56:31+08:00 INFO Harvester started for file: /fd/gameservice2/logs/gs.log
2016-11-01T14:57:01+08:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=208 filebeat.harvester.running=1 publish.events=214 libbeat.logstash.published_and_acked_events=208 registrar.writes=3 libbeat.logstash.publish.read_bytes=48 libbeat.logstash.publish.write_bytes=16003 filebeat.harvester.open_files=1 registrar.states.update=214 libbeat.logstash.call_count.PublishEvents=3 registar.states.current=5 filebeat.harvester.started=1
2016-11-01T14:57:31+08:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=229 libbeat.logstash.publish.read_bytes=24 registrar.writes=3 publish.events=229 registrar.states.update=229 libbeat.logstash.call_count.PublishEvents=3 libbeat.logstash.publish.write_bytes=9747 libbeat.logstash.published_and_acked_events=229
2016-11-01T14:57:51+08:00 ERR Failed to publish events caused by: EOF
2016-11-01T14:57:51+08:00 INFO Error publishing events (retrying): EOF

/etc/filebeat/filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /fd/gameservice2/logs/gs.log*
output.logstash:
  hosts: ["localhost:5044"]

/etc/logstash/conf.d/beats.conf

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => "192.168.199.9:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

Did I miss anything? Thanks in advance.

2

The log shows, that the first time it tried to publish events, it worked and events were acked. So the question is why after that errors appear. Do you see the errors constantly or only from time to time?

3

It seems the error appears every 30 seconds.

2016-11-01T17:50:40+08:00 ERR Failed to publish events caused by: EOF
2016-11-01T17:50:40+08:00 INFO Error publishing events (retrying): EOF
2016-11-01T17:50:45+08:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=98 libbeat.logstash.published_and_acked_events=98 libbeat.logstash.publish.read_errors=2 libbeat.logstash.publish.write_bytes=10261 registrar.writes=3 libbeat.logstash.call_count.PublishEvents=5 libbeat.logstash.published_but_not_acked_events=97 publish.events=98 registrar.states.update=98 libbeat.logstash.publish.read_bytes=30

usually, get an error first, then an info says "retrying", then seconds later a third line which says "published".

PS: I changed logging.level to DEBUG, but did not get any more infomation.

4

Which version of Logstash are you using? Which version of the beats-input-plugin?

5

logstash 5.0.0
filebeat version 5.0.0 (amd64), libbeat 5.0.0

I don't know how to check the version of beats-input-plugin, but I installed it via the script 2 days ago.
./bin/logstash-plugin install logstash-input-beats

6

As it sounds that some of the events come through, you should have a look at your logstash logs if you see there something suspicous.

7

Hi,

I am seeing the same issue. FileBeat > Logstash > ES.

The error was not showed when the first time event was send but now I am getting this in the log file:

2016-11-09T14:57:38Z ERR Failed to publish events caused by: EOF

8

@Maad_Jahangir Please also check if you see some logs on the LS side.

9

This topic was automatically closed after 21 days. New replies are no longer allowed.