Metricbeat Failing to connect to Elasticsearch version 7.1.1 on aws

Hi all,
I am trying to install https://github.com/elastic/helm-charts/tree/master/metricbeat helmchart on kubernetes with the following configurations:

prometheus.yml: |
      metricbeat.modules:
      - module: prometheus
        period: 10s
        hosts: ["10.*.*.*:30048"]
        metrics_path: '/federate'
        query:
          'match[]': '{__name__=~".+"}'
      output.elasticsearch:
        hosts: ["https://************.us-west-2.es.amazonaws.com:443"]

Metricbeat is failing to talk to ES

2019-10-02T19:33:34.765Z	INFO	elasticsearch/client.go:739	Attempting to connect to Elasticsearch version 7.1.1
2019-10-02T19:33:38.383Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260},"total":{"ticks":1720,"time":{"ms":8},"value":1720},"user":{"ticks":1460,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"f840242e-9e2c-4c11-902d-a474ff934612","uptime":{"ms":1200075}},"memstats":{"gc_next":71073296,"memory_alloc":36221184,"memory_total":145617056}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":987},"write":{"bytes":364}},"pipeline":{"clients":4,"events":{"active":4120,"retry":4}}},"system":{"load":{"1":0.87,"15":0.75,"5":0.72,"norm":{"1":0.435,"15":0.375,"5":0.36}}}}}}
2019-10-02T19:34:08.384Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260},"total":{"ticks":1720,"time":{"ms":8},"value":1720},"user":{"ticks":1460,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"f840242e-9e2c-4c11-902d-a474ff934612","uptime":{"ms":1230076}},"memstats":{"gc_next":71073296,"memory_alloc":36676392,"memory_total":146072264}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":4,"events":{"active":4120}}},"system":{"load":{"1":0.66,"15":0.73,"5":0.68,"norm":{"1":0.33,"15":0.365,"5":0.34}}}}}}
2019-10-02T19:34:08.678Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://********.us-west-2.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials
2019-10-02T19:34:08.678Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(https://********.us-west-2.es.amazonaws.com:443)) with 33 reconnect attempt(s)
2019-10-02T19:34:08.678Z	INFO	[publish]	pipeline/retry.go:189	retryer: send unwait-signal to consumer
2019-10-02T19:34:08.678Z	INFO	[publish]	pipeline/retry.go:191	  done
2019-10-02T19:34:08.679Z	INFO	[publish]	pipeline/retry.go:166	retryer: send wait signal to consumer
2019-10-02T19:34:08.685Z	INFO	[publish]	pipeline/retry.go:168	  done	

not sure whats going wrong can some one please help me .
Curl on the metricbeat pod works fine.

@Sanjanar

What version of elasticsearch is this? the AWS fork?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.