Enabling X-Pack in Logstash 7 gives me this error

Elastic Stack Logstash
1

I have a single node ELK 7 stack and I want Kibana to display Logstash in Monitoring. I enabled X-Pack in /etc/logstash/logstash.yml and restart logstash. I see this in /var/log:

Apr 12 11:21:28 nocptc-elk logstash: [2019-04-12T11:21:28,313][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@10.XX.XX.222:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
Apr 12 11:21:28 nocptc-elk logstash: [2019-04-12T11:21:28,348][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.

Later on I see this error:

pr 12 11:25:28 nocptc-elk logstash: [2019-04-12T11:25:28,331][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}

My logstash.yml is:

path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: password
xpack.monitoring.elasticsearch.hosts: ["https://10.XX.XX.222:9200"]

What can I do to troubleshoot this error?

2

That's telling you that it cannot connect. Is elasticsearch running? Verify with "netstat -a | grep 9200" that something is listening to 9200 on that address. If you are binding elasticsearch to 0.0.0.0 then try changing it to the explicit 10.XX.XX.XX address.

3

Nothing is listening but elasticsearch is running:

[root@elk ~]# netstat -a | grep 9200
[root@elk ~]# systemctl status elasticsearch

● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-04-12 11:18:25 CDT; 59min ago
Docs: http://www.elastic.co
Main PID: 48488 (java)
CGroup: /system.slice/elasticsearch.service
├─48488 /usr/share/elasticsearch/jdk/bin/java -Xms16g -Xmx16g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyO...
└─48586 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Apr 12 11:18:25 xx.com systemd[1]: Started Elasticsearch.
Apr 12 11:18:25xx.com elasticsearch[48488]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 a...release.
Hint: Some lines were ellipsized, use -l to show in full.

4

Here's the contents of elasticsearch.yml. Again, this is a single node installation.

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

5

Elasticsearch binds to localhost by default, so I am not surprised that 10.x.x.x address does not work. But it should still show up as listening on the port. Is there anything relavent in the elasticsearch logs?

6

In elasticsearch.log I see:

[2019-04-12T11:18:36,786][INFO ][o.e.c.s.ClusterSettings ] [elk] updating [xpack.monitoring.collection.enabled] from [false] to [true]

That's about it.

7

In /var/log/messages still this:

Apr 12 12:51:05 elk logstash: [2019-04-12T12:51:05,909][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}

8

I fixed it by commenting out the following in /etc/logstash/logstash.yml and restarting logstash:

path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: password
#xpack.monitoring.elasticsearch.hosts: ["https://10.XX.XX.222:9200"]

Logstash now appears in Stack Monitoring on Kibana.

Thank you for the support Badger. Your last entry clued me into what could be the problem.

9

Does 'netstat -an | grep 9200' find it? Maybe the elasticsearch install updates the services database.

10

Yes it does:

[root@elk ~]# netstat -an | grep 9200
tcp 0 0 127.0.0.1:34274 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34302 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34496 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34500 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34296 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34268 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34300 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34314 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34304 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34294 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34292 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34330 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34310 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34298 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34288 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34282 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34494 127.0.0.1:9200 ESTABLISHED
tcp 0 0 127.0.0.1:34306 127.0.0.1:9200 ESTABLISHED
Truncated

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.