Metricbeat Failed to connect EOF

Elastic Stack Beats
1

I am trying to run metricbeat, but no data is showing up in the GUI and I keep getting the following error in the metricbeat logs:

2019-12-06T20:48:49.394Z ERROR pipeline/output.go:100 Failed to connect to backoff(async(tcp://elkurl.com:5040)): EOF
2019-12-06T20:48:49.394Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(async(tcp://elkurl.com:5040)) with 247 reconnect attempt(s)
2019-12-06T20:48:49.394Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-12-06T20:48:49.394Z INFO [publish] pipeline/retry.go:191 done
2019-12-06T20:48:49.394Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-12-06T20:48:49.395Z INFO [publish] pipeline/retry.go:168 done
2019-12-06T20:49:02.718Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1640,"time":{"ms":1}},"total":{"ticks":7500,"time":{"ms":5},"value":7500},"user":{"ticks":5860,"time":{"ms":4}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":4},"info":{"ephemeral_id":"b756f851-24ef-4295-8a36-bdb8ae710fc8","uptime":{"ms":10860061}},"memstats":{"gc_next":47922304,"memory_alloc":24504472,"memory_total":459542688}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"errors":1},"write":{"bytes":170}},"pipeline":{"clients":1,"events":{"active":4117,"retry":181}}},"system":{"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}}}}

Is there somewhere I should check the timeout time? The error occurs every minute or so. Please let me know if there is a way to resolve this. Thank you in advance.

2

Hi @elkuser2!

You are trying to send evens to Logstash right? If so, is your Logstash server reachable?

The timeout option you are asking for is https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#_timeout_2.

Thanks!

3

Hi Chris

I am trying to get the beats (metricbeat, auditbeat, packetbeat, etc) to send events to... Elasticsearch? Correct me if I am wrong, but I believe the beats are a part of Elasticsearch. We are successfully receiving Logstash events. However, we are not receiving events/data for the beats, and all of their logs have the same EOF error.

Thank you

4

From the error you attach I can guess that Metricbeat is configured to send events to Logstash at tcp://elkurl.com:5040. So if you want to send events to Elasticsearch you should use something like this:

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  index: "metricbeat-%{[agent.version]}-%{+yyyy.MM.dd}"

Thanks!

5

Actually, you are correct. I checked our yml files and we are trying to sent the events to Logstash. How can we check if our Logstash server is reachable?
And just to reiterate, we can see on the Discover page of ELK that we are receiving data for Logstash events but not the beats, if this is helpful info.

6

I would suggest telnet command. Something like telnet elkurl.com 5044 from the machine you run Metricbeat.

Also please make sure that the configuration for Logstash output is correct.

Thanks!

7

I am able to connect to the URL using the telnet command. Logstash output configuration also seems correct. We have enabled: true, correct hostname with correct port, SSL cert authority, certificate, and key configured. The port is not blocked.
Thank you

8

This is weird. Could you give more information about your your setup? Do you run on VMs or containers?

Is there anything related in Logstash logs?
Also could you share your configuration?

9

We are running ELK on Linux machines, not containers. I just found an error in logstash-plain.log:

[2019-12-12T12:17:15,986][INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5040, remote: 172.xx.xx.xxx:34300] Handling exception: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE
[2019-12-12T12:17:15,986][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:10000412:SSL routines:OPENSSL_internal:SSLV3_ALERT_BAD_CERTIFICATE

I guess we will need to check our certificates. Let me now what part of the configuration you want to see. Logstash.output?

Thanks

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.