Failed to connect to backoff(elasticsearch(http://10.80.1.220:5044)): read: connection reset by peer

Hi,

I am getting "Connection reset by peer" error while starting the heartbeat. I have tried almost all the solutions given online but none of them worked. Please find the details below:

Elastic Search running on Security Onion 16.04.6.1 - IP: 10.80.1.220
ELK Stack Version: 6.7.2

Host A: 10.80.1.18
Heartbeat version: Tried both 6.7.1 and 7.1 but same result.

Heartbeat Log File:

 2019-06-18T15:16:21.001-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:710	ES Ping(url=http://10.80.1.220:5044)
2019-06-18T15:16:21.052-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:714	Ping request failed with: Get http://110.80.1.220:5044: read tcp 10.80.1.18:58200->10.80.1.220:5044: read: connection reset by peer
2019-06-18T15:16:22.874-0400	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(http://10.80.1.220:5044)): Get http://10.80.1.220:5044: read tcp 10.80.1.18:58200->10.80.1.220:5044: read: connection reset by peer
2019-06-18T15:16:22.874-0400	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(http://10.80.1.220:5044)) with 1 reconnect attempt(s)
2019-06-18T15:16:22.874-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:710	ES Ping(url=http://10.80.1.220:5044)
2019-06-18T15:16:22.932-0400	DEBUG	[elasticsearch]	elasticsearch/client.go:714	Ping request failed with: Get http://10.80.1.220:5044: read tcp 10.80.1.18:58202->10.80.1.220:5044: read: connection reset by peer

Heartbeat.yml file : hosts: ["110.80.1.220:5044"] <------- elastic search and Kibana is commented

#######Security Onion #######

0006_input_beats.conf File content:

input {
  beats {
port => 5044
tags => [ "beat" ]
client_inactivity_timeout => "1200"
  }
}

###########
9500_output_beats.conf File Content:

    filter {
  if "beat" in [tags] {
    mutate {
          ##add_tag => [ "conf_file_9500"]
        }
  }
}
output {
  if "beat" in [tags] {
    elasticsearch {
      hosts => elasticsearch
      index => "logstash-beats-%{+YYYY.MM.dd}"
      template_name => "logstash-beats"
      template => "/beats-template.json"
      template_overwrite => true
    }
  }
}

Logstash Log file:

[2019-06-18T13:58:39,857][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.0.6:5044, remote: 10.80.1.18:57972] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2019-06-18T13:58:39,857][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:392) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:359) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.2.2.jar:?]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.2.2.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
        at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[logstash-input-tcp-5.2.2.jar:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[logstash-input-tcp-5.2.2.jar:?]

Thanks in advance for the help.

Can you share the output section of your heartbeat.yml? It looks like you're trying to talk to logstash with the Elasticsearch protocol

My Elasticseach is commented and I have only included Logstash.

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------

output.elasticsearch:

  # Array of hosts to connect to.

# hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.

  #protocol: "https"

  #username: "elastic"

  #password: "changeme"

#----------------------------- Logstash output --------------------------------

#output.logstash:

  # The Logstash hosts

hosts: ["10.80.1.220:5044"]

Found my error. I didn't remove one "#" from output.logstash. After posting here I saw the error in my configuration file. Thank you so much. I feel so dumb

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.