Question about Heartbeat not being able to communicate with Elastic/Kibana server

Elastic Stack Beats
1

Hello everyone,

I am trying to get familiar with the ELK stack and right now I am encountering a problem. I am using VMare Workstation 15 Pro and have set up an Elastic VM that contains Elasticsearch, Kibana, and Logstash (with Logstash not being used at the moment). I have also set up another Ubuntu server that contains Heartbeat. I am trying to get it so that the Heartbeat server reports to the Elastic server, however I keep getting a connection refused error when running sudo heartbeat test output.

Here is the output:

elasticsearch: http://192.168.31.129:9200...
parse url... OK
dns lookup... OK
addresses: 192.168.31.129
dial up... ERROR dial tcp 192.168.31.129:9200: connect: connection refused

Both machine are running Ubuntu server 18.04. The UFW firewall on the Elastic server is allowing incoming connections to ports 9200 and 9300, however even with the firewall off, I get the same error.

Am I missing something in one of the configs?

Thanks!

2

Have you configured Elasticsearch to bind to an externally accessible IP address instead of the default localhost?

3

Right now in the Network section of the Elastic.yml I have all those areas commented out, this is what it looks like right now:

#network.host: 0.0.0.0
#http.port: 9200

The IP of the Elasticsearch/Kibana server is 192.168.31.129.

4

You need to uncomment the network.host parameter and restart. If not it will be binding to localhost which can not be accessed externally.

5

Ok, I set the config to the following and rebooted the Elastic server:

network.host: 0.0.0.0
#http.port: 9200

However I am getting the same result from the heartbeat test output command on the heartbeat server as well as a Failed status when I check the status of Elasticsearch on the Elastic server I just rebooted.

Trying to get more info now.

6

This is the Elasticsearch service status error:

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Sun 2019-09-22 14:53:56 UTC; 3min 51s ago
Docs: http://www.elastic.co
Process: 1010 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=78)
Main PID: 1010 (code=exited, status=78)

Sep 22 14:53:20 ub-srv-elastic systemd[1]: Started Elasticsearch.
Sep 22 14:53:24 ub-srv-elastic elasticsearch[1010]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sep 22 14:53:56 ub-srv-elastic systemd[1]: elasticsearch.service: Main process exited, code=exited, status=78/n/a
Sep 22 14:53:56 ub-srv-elastic systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

7

Check the Elasticsearch logs. It is possible that you have encountered an issue with the bootstrap checks now that you are binding to a public IP address.

8

I had named my elastic cluster my-test-cluster, so I tailed the my-test-cluster log and this is what I get:

[2019-09-22T14:53:55,735][INFO ][o.e.t.TransportService ] [ub-srv-elastic] publish_address {192.168.31.129:9300}, bound_addresses {[::]:9300}
[2019-09-22T14:53:55,753][INFO ][o.e.b.BootstrapChecks ] [ub-srv-elastic] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-22T14:53:55,800][ERROR][o.e.b.Bootstrap ] [ub-srv-elastic] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2019-09-22T14:53:55,828][INFO ][o.e.n.Node ] [ub-srv-elastic] stopping ...
[2019-09-22T14:53:55,901][INFO ][o.e.n.Node ] [ub-srv-elastic] stopped
[2019-09-22T14:53:55,901][INFO ][o.e.n.Node ] [ub-srv-elastic] closing ...
[2019-09-22T14:53:55,929][INFO ][o.e.n.Node ] [ub-srv-elastic] closed
[2019-09-22T14:53:55,931][INFO ][o.e.x.m.p.NativeController] [ub-srv-elastic] Native controller process has stopped - no new native processes can be started

9

Well, I added the Elastic server's IP and ports 9200 and 9300 to the discovery seed hosts and it looks to be a fix. Elastic status shows as active and I no longer get the heartbeat connection refused error. I'm going to explore this some more to verify but thanks for your help through this.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.