Connection reset by peer when connecting to Elastic

Elastic Stack Beats
,
1

Hi all,

I'm trying to setup Heartbeat without Logstash (directly integrated into Elastic ECK) on a kubernetes platform.
When the pod is starting, I can see the following error message:
Failed to connect to backoff(Elasticsearch(http://elsmonitor-es-transport.elastic-system.svc.cluster.local:9300)): Get "http://elsmonitor-es-transport.elastic-system.svc.cluster.local:9300": read tcp 10.250.0.4:59966->100.96.0.226:9300: read: connection reset by peer

For some reason, the connection fails. I have metricbeat configured and that is working fine. Here is the Elasticsearch output I configured (logstash output is commented out btw):

      output.elasticsearch:
        hosts: ['elsmonitor-es-http.elastic-system.svc.cluster.local:9200']
        username: ${ELASTICSEARCH_USERNAME}
        password: ${ELASTICSEARCH_PASSWORD}
      setup.kibana:
        host: "http://kibmonitor-kb-http.elastic-system.svc.cluster.local:5601"

Any help is appreciated because I tried all kinds of things but without any luck. I also checked the Elastic logs but couldn't find any log back for why the connection was reset.

Bram

2

Hi @brampurnot , connection was reset could encompass a number of issues, all of which are typically network related. Are you sure the pod has connectivity to ES? Is Metricbeat in the same pod? If it was a credentials related issue you'd get a different error.

Unfortunately it's hard to say more than that, it seems that the container heartbeat is running in can't even get as far as a TCP connection. You could also try curl from the same pod to make sure those credentials work.