Failed to connect to backoff(elasticsearch(http://10.80.1.220:5044)): read: connection reset by peer

Elastic Stack Beats
1

Hi,

I am getting "Connection reset by peer" error while starting the heartbeat. I have tried almost all the solutions given online but none of them worked. Please find the details below:

Elastic Search running on Security Onion 16.04.6.1 - IP: 10.80.1.220
ELK Stack Version: 6.7.2

Host A: 10.80.1.18
Heartbeat version: Tried both 6.7.1 and 7.1 but same result.

Heartbeat Log File:

 2019-06-18T15:16:21.001-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:710	ES Ping(url=http://10.80.1.220:5044)
2019-06-18T15:16:21.052-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:714	Ping request failed with: Get http://110.80.1.220:5044: read tcp 10.80.1.18:58200->10.80.1.220:5044: read: connection reset by peer
2019-06-18T15:16:22.874-0400	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://10.80.1.220:5044)): Get http://10.80.1.220:5044: read tcp 10.80.1.18:58200->10.80.1.220:5044: read: connection reset by peer
2019-06-18T15:16:22.874-0400	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://10.80.1.220:5044)) with 1 reconnect attempt(s)
2019-06-18T15:16:22.874-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:710	ES Ping(url=http://10.80.1.220:5044)
2019-06-18T15:16:22.932-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:714	Ping request failed with: Get http://10.80.1.220:5044: read tcp 10.80.1.18:58202->10.80.1.220:5044: read: connection reset by peer

Heartbeat.yml file : hosts: ["110.80.1.220:5044"] <------- elastic search and Kibana is commented

#######Security Onion #######

0006_input_beats.conf File content:

input {
  beats {
port => 5044
tags => [ "beat" ]
client_inactivity_timeout => "1200"
  }
}

###########
9500_output_beats.conf File Content:

    filter {
  if "beat" in [tags] {
    mutate {
          ##add_tag => [ "conf_file_9500"]
        }
  }
}
output {
  if "beat" in [tags] {
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-beats-%{+YYYY.MM.dd}"
      template_name => "logstash-beats"
      template => "/beats-template.json"
      template_overwrite => true
    }
  }
}

Logstash Log file:

[2019-06-18T13:58:39,857][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.0.6:5044, remote: 10.80.1.18:57972] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2019-06-18T13:58:39,857][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.2.2.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[logstash-input-tcp-5.2.2.jar:?]

Thanks in advance for the help.

2

Can you share the output section of your heartbeat.yml? It looks like you're trying to talk to logstash with the Elasticsearch protocol

1 Like
3

My Elasticseach is commented and I have only included Logstash.

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:

  # Array of hosts to connect to.

# hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

hosts: ["10.80.1.220:5044"]
4

Found my error. I didn't remove one "#" from output.logstash. After posting here I saw the error in my configuration file. Thank you so much. I feel so dumb

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.