Elasticsearch Metricbeat Module not parsing hosts properly

Elastic Stack Beats
1

I've enabled the X-Pack on Elasticsearch Module in Metricbeat 7.4.1. I have SSL set up for my Elasticsearch cluster (single node, Kibana, Losgstash, and other beats are fine) and am attempting to get Metricbeat to query Elasticsearch, but it is not parsing the "hosts:" field properly.

Logs state the following:

I've tried a bunch of other combos, but ivariably when I attempt https over port 9200, Metricbeat will attempt to connect to Elasticsearch on http://localhost:9200. To me, this does not jive with what their documentation says I should be able to do.

Is there an error in my config or is there something that Metricbeat is messing up?

Elasticsearch Config:

cluster.routing.allocation.enable: all
node.name: hp-elastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.supported_protocols: TLSv1.3, TLSv1.2, TLSv1.1
xpack.security.http.ssl.key: /etc/elasticsearch/secure/elasticsearch.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/secure/elasticsearch.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/secure/bengel+CA.crt
discovery.type: single-node
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.collection.enabled: true

Metricbeat Config:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

metricbeat.modules:
- module: system
  metricsets:
    - cpu             # CPU usage
    - load            # CPU load averages
    - memory          # Memory usage
    - network         # Network IO
    - process         # Per process metrics
    - process_summary # Process summary
    - uptime          # System Uptime
    - socket_summary  # Socket summary
    - core           # Per CPU core usage
    - diskio         # Disk IO
    - filesystem     # File system usage for each mountpoint
    - fsstat         # File system summary metrics
  enabled: true
  period: 30s
  processes: ['.*']
  cpu.metrics: ["percentages"]
  core.metrics: ["percentages"]
  socket.reverse_lookup.enabled: true
  socket.reverse_lookup.success_ttl: 60s
  socket.reverse_lookup.failure_ttl: 120s

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://elasticsearch.domain.local:9200"]
  xpack.enabled: true

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.dashboards.enabled: true

setup.kibana:
  host: "https://kibana.domain.local:5601"

output.logstash:
  hosts: ["logstash.bengel.local:5044"]
  ssl.certificate_autorities: ["/etc/ssl/certs/ca-bundle.crt"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Log Output

No errors until:

2019-10-30T11:08:22.569-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.ccr: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.570-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.ml_job: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.571-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.index_summary: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.571-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.index_recovery: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.571-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.576-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.node_stats: error making http request: Get http://localhost:9200/_nodes/_local/stats: EOF
2019-10-30T11:08:22.579-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2019-10-30T11:08:22.579-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: EOF
2

Hi @Rad_Engel :slightly_smiling_face:

I'm not fully sure but I think that the error you're getting is when Metricbeat tries to connect to Elasticsearch to send the events it has fetched previosly. Your Elasticsearch host, which is the destination of your Metricbeat events, is just 0.0.0.0

Can you check this?

3

Hi @Mario_Castro,

Thanks for taking a look! I don't think that setting is the issue as other beats are connecting just fine as well as Logstash and I'm able to access the API locally and remotely. Elasticsearch documentation states:

0.0.0.0 is an acceptable IP address and will bind to all network interfaces.

here: Network Settings | Elasticsearch Guide [7.4] | Elastic

Going through and changing the other Filebeat and Metricbeat configs in my environment for SSL and they all parse the 'hosts:' field just fine. It seems that it's an issue with the Metricbeat Elasticsearch module, but I hope that I might be able to work around it with a config tweak.

4

Forgive me, I didn't explained myself properly. I mean that the address should maybe be https://0.0.0.0?

5

Ok, I've now tried https://0.0.0.0:9200 and https://10.5.0.5:9200 with the same behavior.

What's odd is that when something else is wrong (I made a mistake when I regenerated my certificate with more SANs for testing) it first complains that it cannot connect to the configured host, then to http://localhost:9200. So, it's trying both, but only when something else is wrong.

2019-10-31T10:08:50.092-0400    INFO    module/wrapper.go:252   Error fetching data for metricset elasticsearch.shard: error determining if connected Elasticsearch node is master: error making http request: Get https://0.0.0.0:9200/_nodes/_local/nodes: x509: certificate specifies an incompatible key usage

I've fixed the cert issue and no change.

6

Just for completion, which version of Elasticsearch are you using? Metricbeat is 7.4.1 as you already mentioned.

For me it looks like some issue when generating the certs and I know by own experience that it's easy to mess up with them

7

I've just upgraded to Metricbeat 7.4.2 and elasticearch to 7.4.2, no change. I'm very confident that it's not a certificate issue. As mentioned, I fixed the certificate error I made when testing, but that error did shed some light on the way that metricbeat is parsing the elasticsearch monitoring module. I've since enabled user authentication, confirmed that the metricbeat can connect successfully to elasticsearch over https and tested with the 'elastic' superuser without any change. The elasticsearch monitoring module in metricbeat fails to correctly parse the 'hosts:' parameter when there is a combination of 'https' and '9200' in the url and instead uses http://localhost:9200. All other parts of metricbeat, and also filebeat and winlogbeat successfully parse the hostname: https://elasticsearch.domain.local:9200 and connect without issue.

This seems like a bug, but I'm willing to do more testing, but I'll need guidance of where to go from here.

8

Summoning @shaunak that maybe have something to add here. Sorry for the noise if not @shaunak

9

Hi @Rad_Engel,

I just tried to reproduce this scenario locally with Metricbeat 7.4 but was unable to. Could you please post the results of the following command?

metricbeat export config

Thanks,

Shaunak

10

Here's the config and the error that I'm seeing in the logs:

metricbeat:
      config:
        modules:
          path: /etc/metricbeat/modules.d/*.yml
      modules:
      - core:
          metrics:
          - percentages
        cpu:
          petrics:
          - percentages
        enabled: true
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        - uptime
        - socket_summary
        - core
        - diskio
        - filesystem
        - fsstat
        module: system
        period: 30s
        processes:
        - .*
        socket:
          reverse_lookup:
            enabled: true
            failure_ttl: 120s
            success_ttl: 60s
      - enabled: true
        hosts:
        - https://elasticsearch.domain.local:9200
        metricsets:
        - ccr
        - cluster_stats
        - index
        - index_recovery
        - index_summary
        - ml_job
        - node_stats
        - shard
        module: elasticsearch
        password: <password>
        period: 30s
        ssl:
          certificate_autorities:
          - /etc/ssl/certs/ca-bundle.crt
        username: metricbeat
        xpack:
          enabled: true
    monitoring:
      cluster_uuid: UUID
      elasticsearch: null
      enabled: true
    output:
      elasticsearch:
        hosts:
        - https://elasticsearch.domain.local:9200
        password: <password>
        protocol: https
        ssl:
          certificate_autorities:
          - /etc/ssl/certs/ca-bundle.crt
        username: metricbeat
    path:
      config: /etc/metricbeat
      data: /var/lib/metricbeat
      home: /usr/share/metricbeat
      logs: /var/log/metricbeat
    processors:
    - add_host_metadata: null
    setup:
      dashboards:
        enabled: true
      ilm:
        enabled: false
      kibana:
        host: https://kibana.domain.local
        protocol: https

Nov 14 13:21:22 elasticsearch metricbeat: 2019-11-14T13:21:22.113-0500#011INFO#011module/wrapper.go:252#011Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get http://localhost:9200/_nodes/_local/nodes: dial tcp 127.0.0.1:9200: connect: connection refused

11

Thanks. Could you post the output of the following?

ls -l /etc/metricbeat/modules.d/ | grep elasticsearch

Thanks,

Shaunak

12 
-rw-r--r--. 1 root root  400 Oct 22 12:25 elasticsearch-xpack.yml
-rw-r--r--. 1 root root  400 Oct 28 15:49 elasticsearch-xpack.yml.disabled
-rw-r--r--. 1 root root  283 Oct 28 15:49 elasticsearch.yml.disabled
13

Thanks. Now could you paste the output of the following?

cat /etc/metricbeat/modules.d/elasticsearch-xpack.yml

Shaunak

14 
# Module: elasticsearch
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.4/metricbeat-module-elasticsearch.html

- module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["http://localhost:9200"]
  #username: "user"
  #password: "secret"
  xpack.enabled: true
15

Thanks. I believe here's what's going on:

In your Metricbeat configuration file you have these lines:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

That is causing all enabled module configurations from your /etc/metricbeat/modules.d/ folder to be read and parsed by Metricbeat. One of the module configurations in this folder is elasticsearch-xpack.yml, which contains the line:

  hosts: ["http://localhost:9200"]

I believe that's where the error in the logs is coming from.

You have two options on how to fix this:

  1. In your Metricbeat configuration file you have the following section:

    - module: elasticsearch
  metricsets:
    - ccr
    - cluster_stats
    - index
    - index_recovery
    - index_summary
    - ml_job
    - node_stats
    - shard
  period: 10s
  hosts: ["https://elasticsearch.domain.local:9200"]
  xpack.enabled: true

    You could comment out or delete this section, edit /etc/metricbeat/modules.d/elasticsearch-xpack.yml and set the hosts field in there as:

    hosts: [ "https://elasticsearch.domain.local:9200" ]

  2. Alternatively, you could leave your Metricbeat configuration as it is right now, but remove the /etc/metricbeat/modules.d/elasticsearch-xpack.yml file. You can either do this remove manually or by running metricbeat modules disable elasticsearch-xpack.

Please try one of these options and let me know if the errors in the log go away.

Thanks,

Shaunak

1 Like
16

@shaunak That was it. Thanks so much for your help!

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.