# Elasticsearch Metricbeat Module not parsing hosts properly **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [October 30, 2019, 3:20pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893 "2019-10-30T15:20:36Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [October 30, 2019, 3:20pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/1 "2019-10-30T15:20:36Z")
I've enabled the X-Pack on Elasticsearch Module in Metricbeat 7.4.1. I have SSL set up for my Elasticsearch cluster (single node, Kibana, Losgstash, and other beats are fine) and am attempting to get Metricbeat to query Elasticsearch, but it is not parsing the "hosts:" field properly. Logs state the following: - As expected a "hosts:" of "[http://localhost:9200](http://localhost:9200)", Metricbeat attempts to contact [http://localhost:9200](http://localhost:9200) - hosts: "[http://elasticsearch.domain.local:9200](http://elasticsearch.domain.local:9200)" -\> [http://elasticsearch.domain.local:9200](http://elasticsearch.domain.local:9200) - But, hosts: "[https://localhost:9200](https://localhost:9200)" -\> [http://localhost:9200](http://localhost:9200) - and, hosts: "[https://elasticsearch.domain.local:9200](https://elasticsearch.domain.local:9200)" -\> [http://localhost:9200](http://localhost:9200) I've tried a bunch of other combos, but ivariably when I attempt https over port 9200, Metricbeat will attempt to connect to Elasticsearch on [http://localhost:9200](http://localhost:9200). To me, this does not jive with what their documentation says I should be able to do. Is there an error in my config or is there something that Metricbeat is messing up? # Elasticsearch Config: ``` cluster.routing.allocation.enable: all node.name: hp-elastic path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 http.port: 9200 xpack.security.http.ssl.enabled: true xpack.security.http.ssl.supported_protocols: TLSv1.3, TLSv1.2, TLSv1.1 xpack.security.http.ssl.key: /etc/elasticsearch/secure/elasticsearch.key xpack.security.http.ssl.certificate: /etc/elasticsearch/secure/elasticsearch.crt xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/secure/bengel+CA.crt discovery.type: single-node xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.collection.enabled: true ``` # Metricbeat Config: ``` metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true reload.period: 10s metricbeat.modules: - module: system metricsets: - cpu # CPU usage - load # CPU load averages - memory # Memory usage - network # Network IO - process # Per process metrics - process_summary # Process summary - uptime # System Uptime - socket_summary # Socket summary - core # Per CPU core usage - diskio # Disk IO - filesystem # File system usage for each mountpoint - fsstat # File system summary metrics enabled: true period: 30s processes: ['.*'] cpu.metrics: ["percentages"] core.metrics: ["percentages"] socket.reverse_lookup.enabled: true socket.reverse_lookup.success_ttl: 60s socket.reverse_lookup.failure_ttl: 120s - module: elasticsearch metricsets: - ccr - cluster_stats - index - index_recovery - index_summary - ml_job - node_stats - shard period: 10s hosts: ["https://elasticsearch.domain.local:9200"] xpack.enabled: true setup.template.settings: index.number_of_shards: 1 index.codec: best_compression setup.dashboards.enabled: true setup.kibana: host: "https://kibana.domain.local:5601" output.logstash: hosts: ["logstash.bengel.local:5044"] ssl.certificate_autorities: ["/etc/ssl/certs/ca-bundle.crt"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~ ``` # Log Output No errors until: ``` 2019-10-30T11:08:22.569-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.ccr: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.570-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.571-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.571-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.index_recovery: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.571-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.576-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.node_stats: error making http request: Get http://localhost:9200/_nodes/_local/stats: EOF 2019-10-30T11:08:22.579-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF 2019-10-30T11:08:22.579-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF ``` ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [October 31, 2019, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/2 "2019-10-31T10:10:47Z")
Hi @Rad_Engel 🙂 I'm not fully sure but I think that the error you're getting is when Metricbeat tries to connect to Elasticsearch to send the events it has fetched previosly. Your Elasticsearch host, which is the destination of your Metricbeat events, is just `0.0.0.0` Can you check this? ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [October 31, 2019, 11:53am UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/3 "2019-10-31T11:53:33Z")
Hi @Mario_Castro, Thanks for taking a look! I don't think that setting is the issue as other beats are connecting just fine as well as Logstash and I'm able to access the API locally and remotely. Elasticsearch documentation states: > `0.0.0.0` is an acceptable IP address and will bind to all network interfaces. here: [Network Settings | Elasticsearch Guide [7.4] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.4/modules-network.html) Going through and changing the other Filebeat and Metricbeat configs in my environment for SSL and they all parse the 'hosts:' field just fine. It seems that it's an issue with the Metricbeat Elasticsearch module, but I hope that I might be able to work around it with a config tweak. ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [October 31, 2019, 1:06pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/4 "2019-10-31T13:06:16Z")
Forgive me, I didn't explained myself properly. I mean that the address should maybe be `https://0.0.0.0`? ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [October 31, 2019, 2:22pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/5 "2019-10-31T14:22:29Z")
Ok, I've now tried `https://0.0.0.0:9200` and `https://10.5.0.5:9200` with the same behavior. What's odd is that when something else is wrong (I made a mistake when I regenerated my certificate with more SANs for testing) it first complains that it cannot connect to the configured host, then to `http://localhost:9200`. So, it's trying both, but only when something else is wrong. ``` 2019-10-31T10:08:50.092-0400 INFO module/wrapper.go:252 Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get https://0.0.0.0:9200/_nodes/_local/nodes: x509: certificate specifies an incompatible key usage ``` I've fixed the cert issue and no change. ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [November 1, 2019, 9:07pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/6 "2019-11-01T21:07:35Z")
Just for completion, which version of Elasticsearch are you using? Metricbeat is 7.4.1 as you already mentioned. For me it looks like some issue when generating the certs and I know by own experience that it's easy to mess up with them ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [November 4, 2019, 7:49pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/7 "2019-11-04T19:49:29Z")
I've just upgraded to Metricbeat 7.4.2 and elasticearch to 7.4.2, no change. I'm very confident that it's not a certificate issue. As mentioned, I fixed the certificate error I made when testing, but that error did shed some light on the way that metricbeat is parsing the elasticsearch monitoring module. I've since enabled user authentication, confirmed that the metricbeat can connect successfully to elasticsearch over https and tested with the 'elastic' superuser without any change. The elasticsearch monitoring module in metricbeat fails to correctly parse the 'hosts:' parameter when there is a combination of 'https' and '9200' in the url and instead uses [http://localhost:9200](http://localhost:9200). All other parts of metricbeat, and also filebeat and winlogbeat successfully parse the hostname: [https://elasticsearch.domain.local:9200](https://elasticsearch.domain.local:9200) and connect without issue. This seems like a bug, but I'm willing to do more testing, but I'll need guidance of where to go from here. ---
**Author:** ![Mario\_Castro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mario_castro/32/35107_2.png) [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)\ **Post date:** [November 12, 2019, 12:10pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/8 "2019-11-12T12:10:49Z")
Summoning @shaunak that maybe have something to add here. Sorry for the noise if not @shaunak ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2019, 4:59pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/9 "2019-11-14T16:59:13Z")
Hi @Rad_Engel, I just tried to reproduce this scenario locally with Metricbeat 7.4 but was unable to. Could you please post the results of the following command? ```auto metricbeat export config ``` Thanks, Shaunak ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [November 14, 2019, 6:26pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/10 "2019-11-14T18:26:47Z")
Here's the config and the error that I'm seeing in the logs: ``` metricbeat: config: modules: path: /etc/metricbeat/modules.d/*.yml modules: - core: metrics: - percentages cpu: petrics: - percentages enabled: true metricsets: - cpu - load - memory - network - process - process_summary - uptime - socket_summary - core - diskio - filesystem - fsstat module: system period: 30s processes: - .* socket: reverse_lookup: enabled: true failure_ttl: 120s success_ttl: 60s - enabled: true hosts: - https://elasticsearch.domain.local:9200 metricsets: - ccr - cluster_stats - index - index_recovery - index_summary - ml_job - node_stats - shard module: elasticsearch password: period: 30s ssl: certificate_autorities: - /etc/ssl/certs/ca-bundle.crt username: metricbeat xpack: enabled: true monitoring: cluster_uuid: UUID elasticsearch: null enabled: true output: elasticsearch: hosts: - https://elasticsearch.domain.local:9200 password: protocol: https ssl: certificate_autorities: - /etc/ssl/certs/ca-bundle.crt username: metricbeat path: config: /etc/metricbeat data: /var/lib/metricbeat home: /usr/share/metricbeat logs: /var/log/metricbeat processors: - add_host_metadata: null setup: dashboards: enabled: true ilm: enabled: false kibana: host: https://kibana.domain.local protocol: https ``` > Nov 14 13:21:22 elasticsearch metricbeat: 2019-11-14T13:21:22.113-0500#011INFO#011module/wrapper.go:252#011Error fetching data for metricset elasticsearch.cluster\_stats: error determining if connected Elasticsearch node is master: error making http request: Get [http://localhost:9200/\_nodes/\_local/nodes:](http://localhost:9200/_nodes/_local/nodes:) dial tcp 127.0.0.1:9200: connect: connection refused ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2019, 6:42pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/11 "2019-11-14T18:42:58Z")
Thanks. Could you post the output of the following? ```auto ls -l /etc/metricbeat/modules.d/ | grep elasticsearch ``` Thanks, Shaunak ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [November 14, 2019, 7:08pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/12 "2019-11-14T19:08:52Z")
``` -rw-r--r--. 1 root root 400 Oct 22 12:25 elasticsearch-xpack.yml -rw-r--r--. 1 root root 400 Oct 28 15:49 elasticsearch-xpack.yml.disabled -rw-r--r--. 1 root root 283 Oct 28 15:49 elasticsearch.yml.disabled ``` ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2019, 7:10pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/13 "2019-11-14T19:10:19Z")
Thanks. Now could you paste the output of the following? ```auto cat /etc/metricbeat/modules.d/elasticsearch-xpack.yml ``` Shaunak ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [November 14, 2019, 7:13pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/14 "2019-11-14T19:13:00Z")
``` # Module: elasticsearch # Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.4/metricbeat-module-elasticsearch.html - module: elasticsearch metricsets: - ccr - cluster_stats - index - index_recovery - index_summary - ml_job - node_stats - shard period: 10s hosts: ["http://localhost:9200"] #username: "user" #password: "secret" xpack.enabled: true ``` ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [November 14, 2019, 7:19pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/15 "2019-11-14T19:19:25Z")
Thanks. I believe here's what's going on: In your Metricbeat configuration file you have these lines: ```auto metricbeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true reload.period: 10s ``` That is causing all enabled module configurations from your `/etc/metricbeat/modules.d/` folder to be read and parsed by Metricbeat. One of the module configurations in this folder is `elasticsearch-xpack.yml`, which contains the line: ```auto hosts: ["http://localhost:9200"] ``` I believe that's where the error in the logs is coming from. You have two options on how to fix this: 1. In your Metricbeat configuration file you have the following section: 2. Alternatively, you could leave your Metricbeat configuration as it is right now, but remove the `/etc/metricbeat/modules.d/elasticsearch-xpack.yml` file. You can either do this remove manually or by running `metricbeat modules disable elasticsearch-xpack`. Please try one of these options and let me know if the errors in the log go away. Thanks, Shaunak ---
**Author:** ![Rad\_Engel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rad_engel/32/50344_2.png) [@Rad\_Engel](https://discuss.elastic.co/u/Rad_Engel)\ **Post date:** [November 14, 2019, 7:43pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/16 "2019-11-14T19:43:34Z")
@shaunak That was it. Thanks so much for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2019, 7:43pm UTC](https://discuss.elastic.co/t/elasticsearch-metricbeat-module-not-parsing-hosts-properly/205893/17 "2019-12-12T19:43:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.