Heartbeat showing 1/2 of my nodes down

Elastic Stack Beats
1

Hey I have two nodes of elasticsearch running in two separate VMs in azure, I set up SSL + https for my cluster and secured kibana too running in VM1. I used self signed certificates following this doc. I used the file elasticsearch-ca.pem generated in the Encrypt HTTP client communications for Elasticsearch in my heartbeat.yml file.

I installed heartbeat in VM1 and VM2, for VM1 heartbeat showing a status of Up, whereas for VM2 showing down with the following error:

Get "https://localhost:9200": http: server gave HTTP response to HTTPS client]

my heartbeat.yml for vm2 (similar to my conf in vm1):

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: monitor-node2
  # Human readable display name for this service in Uptime UI and elsewhere
  name: Monitor node2
 # List or urls to query
  urls: ["https://localhost:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  # Configure task schedule
  schedule: '@every 10s'
  # Total test connection and data exchange timeout
  #timeout: 16s
  # Name of corresponding APM service, if Elastic APM is in use for the monitored service.
  #service.name: my-apm-service-name
  ssl:
    certificate_authorities: ['/etc/heartbeat/config/certs/elasticsearch-ca.pem']
    verification_mode: none
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
        host: "https://nodeName.regionName.cloudapp.azure.com:5601"
        ssl.enabled: true
        ssl.verification_mode: none

output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["node1:9200", "node2:9200"]
        username: "elastic"
        password: "${ES_PWD}"
        protocol: "https"
        ssl.verification_mode: none

processors:
  - add_observer_metadata:

heartbeat.yml for vm1:

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: monitor-demo-elastic-node1
  # Human readable display name for this service in Uptime UI and elsewhere
  name: Monitor node1
  # List or urls to query
  urls: ["https://localhost:9200"]
  username: "elastic"
  password: "${ES_PWD}"
  # Configure task schedule
  schedule: '@every 10s'
  ssl:
    certificate_authorities: ['/etc/heartbeat/config/certs/elasticsearch-ca.pem']
    verification_mode: none
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
        host: "localhost:5601"
        ssl.enabled: true
        ssl.verification_mode: none

output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["node1:9200", "node2:9200"]
        username: "elastic"
        password: "${ES_PWD}"
        protocol: "https"
        ssl.verification_mode: none
processors:
  - add_observer_metadata:

one thing to mention that I have the same error in metricbeat for vm2 (aka node2):

metricbeat[12848]: 2021-04-29T16:19:13.321Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://node2:9200)): Get "https://node2:9200": http: server gave HTTP response to HTTPS client

I don't know what's wrong is it a problem related to elasticsearch-ca.pem file ?

2

It seems like ES is not setup to use TLS based on that error. Have you confirmed that using curl or a browser ES is actually setup to use TLS?

3

yes @Andrew_Cholakian1 thanks for your comment, as you said I checked my elasticsearch.yml in vm2 and found that the following two lines were commented:

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/config/certs/http.p12

after uncommenting the above lines problem resolved! :grinning:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.