# Unable to get Metricbeat to communicate with Elasticsearch **URL:** \ **Category:** Beats\ **Tags:** docker, metricbeat\ **Created:** [November 8, 2023, 11:33pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747 "2023-11-08T23:33:34Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 8, 2023, 11:33pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/1 "2023-11-08T23:33:34Z")
I'm working on upgrading the Elastic stack to the current version for my company. So, I'm testing the deployment of Elasticsearch, Kibana, Logstash, and Metricbeat. Each component is in a separate docker container but all on the same host. I believe I've successfully configured the first three components, and now I'm attempting to get Metricbeat to communicate with Elasticsearch. Unfortunately, I receive one of two errors no matter what I do. 1. `Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get \"http://localhost:9200/_nodes/_local/nodes\": dial tcp [::1]:9200: connect: cannot assign requested address` 2. `Failed to connect to backoff(elasticsearch(http://10.19.12.43:9200)): Get \"http://10.19.12.43:9200\` I've reviewed [this post](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893), and if I disable the elasticsearch-xpack while leaving metricbeat.yml as it is, I receive the second error above. At this point, I'm running in circles. So, any assistance would be most appreciated. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 8, 2023, 11:44pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/2 "2023-11-08T23:44:43Z")
What version? How did you install elasticsearch? What are you trying to accomplish with metricbeat send system / host metrics or monitor elasticsearch? Try running this and show the output. `metricbeat test output` Please share the `output` section or your metricbeat.yml ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 12:31am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/3 "2023-11-09T00:31:13Z")
Hi Stephen, thanks for your reply. I pulled the standard Elastic docker containers. They are all 8.10.4 Below is information returned from `metricbeat test output`. ```auto elasticsearch: https://10.19.12.43:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 10.19.12.43 dial up... OK TLS... security... WARN server's certificate chain verification is disabled handshake... OK TLS version: TLSv1.3 dial up... OK talk to server... OK version: 8.10.4 ``` from metricbeat.yml ```auto output.elasticsearch: hosts: ["https://10.19.12.43:9200"] username: password: ssl: enabled: true verification_mode: none ca_trusted_fingerprint: "AD553FA4B1A62094E49CF061421059379ABCCF671493A8F6759DF52D45E30E77" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 2:01am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/4 "2023-11-09T02:01:47Z")
Hi @james_fourth So that is good so far. BTW if that is a valid finger print you can take out / comment out > [@james\_fourth](#): > > ` verification_mode: none` The post you looked at is incredibly old... so I would not necessarily use that. So, back to what are you trying to collect / accomplish with metricbeat did you enable any modules? It looks like you enabled the `elasticsearch` moule or `elasticsearch-xpack` module... those are used to monitor elasticsearch... is that what you are trying to do? ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 2:18am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/5 "2023-11-09T02:18:47Z")
Thanks for the tip about the fingerprint, and I’m glad to hear the output of the test looks as good to you as it did to me. I also wondered about the age of the post I referenced, and I hadn’t seen a more recent post that appeared as relevant to the issue I’m seeing. You are correct. I am trying to monitor elasticsearch. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 2:21am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/6 "2023-11-09T02:21:50Z")
> [@james\_fourth](#): > > You are correct. I am trying to monitor elasticsearch. The Same Elasticsearch or a different one. You should read > **[Collecting Elasticsearch monitoring data with Metricbeat | Elasticsearch...](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-metricbeat.html)** ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 2:40am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/7 "2023-11-09T02:40:57Z")
The same elasticsearch. I’ve read that doc, but maybe I could read through it more carefully. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 3:06am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/8 "2023-11-09T03:06:20Z")
Post your `elasticsearch-xpack.yml` NOTE do not have both `elasticsearch-xpack.yml` and `elasticsearch.yml` Enabled they will overwrite each other you can also run `metricbeat test config` Your error shows `Get \"http://localhost:9200/_nodes/_local/nodes` But your elasticsearch is running on `https` so you need to use https in the elasticsearch-xpack and provide the CA just like your output section ```auto - module: elasticsearch xpack.enabled: true period: 10s hosts: ["https://10.19.12.43:9200"] username: password: ssl: enabled: true ca_trusted_fingerprint: "AD553FA4B1A62094E49CF061421059379ABCCF671493A8F6759DF52D45E30E77" ``` ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 6:51pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/9 "2023-11-09T18:51:55Z")
elasticsearch-xpack.yml ```auto - module: elasticsearch xpack.enabled: true period: 10s hosts: ["https://10.19.12.43:9200"] ``` with the remote\_monitoring\_user credentials below the hosts array. The elasticsearch module isn't enabled, only elasticsearch-xpack. When I'm interacting with the metricbeat container, I don't see a filebeat executable available. Is that something I'm missing? ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 7:02pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/10 "2023-11-09T19:02:29Z")
After removing the verification\_mode line, I receive a handshake error after executing `metricbeat test output` saying `certificate is valid for 172.22.0.2, 127.0.0.1, not 10.19.12.43`. Updated metricbeat.yml to refer to localhost instead of 10.19.12.43, and now the dial up test returns an error saying `dial tcp 127.0.0.1:9200: connect: connection refused`. ```auto elasticsearch: https://localhost:9200... parse url... OK connection... parse host... OK dns lookup... OK addresses: 127.0.0.1, ::1 dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused ``` Updated the elasticsearch-xpack.yml to have localhost instead of 10.19.12.43 and restarted the container. However, `metricbeat test output` now returns a dial up error of `dial tcp [::1]:9200: connect: cannot assign requested address` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 7:15pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/11 "2023-11-09T19:15:44Z")
**So in short just put the exact same connection credential in the elasticsearch-xpack module... first and try that** and try with the `elastic` super user. You have a number of things that are causing issues. OK so you need to put this back in ` verification_mode: none` > [@james\_fourth](#): > > `certificate is valid for 172.22.0.2, 127.0.0.1, not 10.19.12.43`. Because you are using different IPs to access elasticsearch than what was used when you created the self signed cert. Thus there is a certificate mismatch.... you can fix that later. > [@james\_fourth](#): > > localhost instead of 10.19.12.43, and now the dial up test returns an error saying `dial tcp 127.0.0.1:9200: connect: connection refused`. Looks like your elasticsearch is not actually running on localhost / loopback because it has been bound to a network interface.... on your host these are separate interfaces > [@james\_fourth](#): > > with the **remote\_monitoring\_user** credentials below the hosts array. Are you sure that use is setup correctly first I would try with the `elastic ` superuser then try the `remote_monitoring_user` > [@james\_fourth](#): > > When I'm interacting with the metricbeat container, I don't see a filebeat executable available. Is that something I'm missing? I do not know what you mean ... clearly you are running `metricbeat test output` ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 7:29pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/12 "2023-11-09T19:29:52Z")
Restored `verification_mode: none` `docker ps` outputs the following ```auto aef6b5ab2b24 docker.elastic.co/beats/metricbeat:8.10.4 "/usr/bin/tini -- /u…" 20 hours ago Up 7 minutes mb01 6ca05c0d6919 docker.elastic.co/logstash/logstash:8.10.4 "/usr/local/bin/dock…" 2 days ago Up 21 hours 0.0.0.0:5400->5400/tcp, 5044/tcp, 0.0.0.0:9600->9600/tcp log01 3d68776452cb docker.elastic.co/kibana/kibana:8.10.4 "/bin/tini -- /usr/l…" 2 days ago Up 21 hours 0.0.0.0:5601->5601/tcp kib01 f9fc41fa6724 docker.elastic.co/elasticsearch/elasticsearch:8.10.4 "/bin/tini -- /usr/l…" 2 days ago Up 21 hours 0.0.0.0:9200->9200/tcp, 9300/tcp es01 ``` So, does that mean I should use `0.0.0.0` for everything? I was previously using the elastic superuser, I'll revert to using those credentials. I'm not sure the `remote_monitoring_user` is set up correctly. I assumed it was configured by default for this kind of operation. I can execute `metricbeat test output`, but if I try `filebeat test output` as mentioned a couple replies back, the command isn't found. Maybe that was a typo? Still receiving connection refused with the elastic superuser, verification\_mode: none, and an address of 0.0.0.0. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 7:36pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/13 "2023-11-09T19:36:49Z")
> [@james\_fourth](#): > > I can execute `metricbeat test output`, but if I try `filebeat test output` as mentioned a couple replies back, the command isn't found. Maybe that was a typo? Sorry my typo I meant `metricbeat` ... fixed > [@james\_fourth](#): > > So, does that mean I should use `0.0.0.0` for everything? > > I was previously using the elastic superuser, I'll revert to using those credentials. No Clue I don't know how you are running these in Compose with Shared Network or individual containers..swarm... etc...?? The ips / what to bind too... you need to understand your environment, This is probably more of a Docker / Networking thing... Are you following some 3rd party article? > [@james\_fourth](#): > > I'm not sure the `remote_monitoring_user` is set up correctly. I assumed it was configured by default for this kind of operation. the roles are created but not the user, I am unclear what you have done... I am only getting pieces of info ... ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 9, 2023, 7:42pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/14 "2023-11-09T19:42:51Z")
Pardon me for not being comprehensive. I'm not using compose or swarm. I'm simply following the Elastic docs pertaining to Docker for each of the Elastic components and using the commands the docs contain. When pulling the containers and using them as default, there are already a collection of users with roles assigned. ![CleanShot 2023-11-09 at 12.41.34](https://us1.discourse-cdn.com/elastic/original/3X/8/f/8f6c633bfce849a68f19c2f4d311c779bc763cb2.png) > **[Install Kibana with Docker | Kibana Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/kibana/current/docker.html)** > **[Running Logstash on Docker | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/docker.html)** > **[Run Metricbeat on Docker | Metricbeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html)** ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 9, 2023, 8:09pm UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/15 "2023-11-09T20:09:55Z")
Ok so when running separate docker containers each container is its own `localhost` so referencing `localhost` in the metricbeat container is not the `localhost` in the elasticsearch container. This is docker stuff. Perhaps look at > [@Filebeat Docker - Elasticsearch Host Issue](https://discuss.elastic.co/t/filebeat-docker-elasticsearch-host-issue/320083/2): > > Hi @mwsprotte Welcome to the community! Docker Networking Issue not Elastic... Another explanation Might be worth reading... if you are new to docker try using host.docker.internal docker run \ --net=elastic docker.elastic.co/beats/filebeat:7.17.4 \ setup -E setup.kibana.host=host.docker.internal:5601 \ -E output.elasticsearch.hosts=["https://host.docker.internal:9200"] So when you want one container to communicate/connect with another container you need to use either the IP of the host or use `host.docker.internal` ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 10, 2023, 12:36am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/17 "2023-11-10T00:36:12Z")
Thanks for the reminder of a docker container being a system unto itself. With that in mind, of course localhost would refer to the container and not the host environment. Unfortunately, I'm using a Oracle Linux 7.9 as the host and the most recent version of docker available for this distro is 19.03. That means `host.docker.internal` isn't available. I attempted to use the host IP of 10.19.12.43 in the elasticsearch-xpack.yml and metricbeat.yml files, but I'm still receiving errors saying `connection refused` and `error determining if connected Elasticsearch node is master`. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [November 10, 2023, 1:26am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/18 "2023-11-10T01:26:05Z")
How many elasticsearch nodes? Are you running just one ? The connection refused is the key message after that all the rest of them are just error messages when it can't connect so obviously it can't determine the node type. So how you debug ....,`exec` into the metricbeat container and try to curl the the elasticsearch using the same URL and credentials that you're using in the xpack What credentials are you using in the X pack? ---
**Author:** ![james\_fourth](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_fourth/32/127398_2.png) [@james\_fourth](https://discuss.elastic.co/u/james_fourth)\ **Post date:** [November 10, 2023, 1:47am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/19 "2023-11-10T01:47:01Z")
I'm running a single Elasticsearch node. I'm using the elastic superuser credentials in the xpack modules. I think I found the mistake. I hadn't added the ssl section to the elasticsearch-xpack like this... ```auto ssl: enabled: true verication_mode: none ca_trusted_fingerprint: "AD553FA4B1A62094E49CF061421059379ABCCF671493A8F6759DF52D45E30E77" ``` Now I'm seeing successful communication with Elasticsearch! ```auto {"log.level":"info","@timestamp":"2023-11-10T01:32:59.859Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":221665564}},"memory":{"mem":{"usage":{"bytes":53698560}}}},"cpu":{"system":{"ticks":230,"time":{"ms":60}},"total":{"ticks":860,"time":{"ms":220},"value":860},"user":{"ticks":630,"time":{"ms":160}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":33},"info":{"ephemeral_id":"1889797d-aa35-4b20-aa5c-675ecf22f8bf","uptime":{"ms":90113},"version":"8.10.4"},"memstats":{"gc_next":29871416,"memory_alloc":18787440,"memory_total":103813872,"rss":120987648},"runtime":{"goroutines":165}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":246,"active":0,"batches":15,"duplicates":108,"total":354},"read":{"bytes":109852},"write":{"bytes":689262}},"pipeline":{"clients":23,"events":{"active":1,"published":354,"total":354},"queue":{"acked":354}}},"metricbeat":{"docker":{"container":{"events":12,"success":12},"cpu":{"events":12,"success":12},"diskio":{"events":12,"success":12},"info":{"events":3,"success":3},"memory":{"events":12,"success":12},"network":{"events":12,"success":12}},"elasticsearch":{"cluster_stats":{"events":3,"success":3},"enrich":{"events":3,"success":3},"index":{"events":105,"success":105},"index_recovery":{"events":45,"success":45},"index_summary":{"events":3,"success":3},"node_stats":{"events":3,"success":3},"shard":{"events":108,"success":108}},"kibana":{"cluster_actions":{"events":3,"failures":3},"cluster_rules":{"events":3,"failures":3},"node_actions":{"events":3,"failures":3},"node_rules":{"events":3,"failures":3},"stats":{"events":3,"failures":3}},"logstash":{"node":{"events":3,"failures":3},"node_stats":{"events":3,"failures":3}}},"system":{"load":{"1":0.01,"15":0.15,"5":0.13,"norm":{"1":0.005,"15":0.075,"5":0.065}}}},"ecs.version":"1.6.0"}} ``` I'm also trying to monitor Kibana and Logstash, and there are still errors from those two like the following... ```auto {"log.level":"error","@timestamp":"2023-11-10T01:32:59.873Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: error making http request: Get \"https://172.22.0.4:9600/\": http: server gave HTTP response to HTTPS client","service.name":"metricbeat","ecs.version":"1.6.0"} {"log.level":"error","@timestamp":"2023-11-10T01:32:59.874Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.cluster_rules: error making http request: Get \"https://172.22.0.3:5601/api/status\": http: server gave HTTP response to HTTPS client","service.name":"metricbeat","ecs.version":"1.6.0"} ``` That said, it does appear progress is being made because I can see Elasticsearch monitored with Metricbeat in Stack Monitoring. ![CleanShot 2023-11-09 at 18.49.14](https://us1.discourse-cdn.com/elastic/original/3X/f/a/faaf0318e4ae9373d0daa450bd7a052a2647bdcd.png) Thank you for your patience and attention. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2023, 3:48am UTC](https://discuss.elastic.co/t/unable-to-get-metricbeat-to-communicate-with-elasticsearch/346747/20 "2023-12-08T03:48:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.