Hi all,
I am trying to install https://github.com/elastic/helm-charts/tree/master/metricbeat helmchart on kubernetes with the following configurations:
prometheus.yml: |
metricbeat.modules:
- module: prometheus
period: 10s
hosts: ["10.*.*.*:30048"]
metrics_path: '/federate'
query:
'match[]': '{__name__=~".+"}'
output.elasticsearch:
hosts: ["https://************.us-west-2.es.amazonaws.com:443"]
Metricbeat is failing to talk to ES
2019-10-02T19:33:34.765Z INFO elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 7.1.1
2019-10-02T19:33:38.383Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260},"total":{"ticks":1720,"time":{"ms":8},"value":1720},"user":{"ticks":1460,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"f840242e-9e2c-4c11-902d-a474ff934612","uptime":{"ms":1200075}},"memstats":{"gc_next":71073296,"memory_alloc":36221184,"memory_total":145617056}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":987},"write":{"bytes":364}},"pipeline":{"clients":4,"events":{"active":4120,"retry":4}}},"system":{"load":{"1":0.87,"15":0.75,"5":0.72,"norm":{"1":0.435,"15":0.375,"5":0.36}}}}}}
2019-10-02T19:34:08.384Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260},"total":{"ticks":1720,"time":{"ms":8},"value":1720},"user":{"ticks":1460,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"f840242e-9e2c-4c11-902d-a474ff934612","uptime":{"ms":1230076}},"memstats":{"gc_next":71073296,"memory_alloc":36676392,"memory_total":146072264}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":4,"events":{"active":4120}}},"system":{"load":{"1":0.66,"15":0.73,"5":0.68,"norm":{"1":0.33,"15":0.365,"5":0.34}}}}}}
2019-10-02T19:34:08.678Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://********.us-west-2.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials
2019-10-02T19:34:08.678Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://********.us-west-2.es.amazonaws.com:443)) with 33 reconnect attempt(s)
2019-10-02T19:34:08.678Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-10-02T19:34:08.678Z INFO [publish] pipeline/retry.go:191 done
2019-10-02T19:34:08.679Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-10-02T19:34:08.685Z INFO [publish] pipeline/retry.go:168 done
not sure whats going wrong can some one please help me .
Curl on the metricbeat pod works fine.