Logstash Instance Missing from Kibana Monitoring

Elastic Stack Logstash
1

I've configured Filebeat to output sample logs to Logstash. Through Logstash, I've set my *.conf file to output the data to Elasticsearch. My end goal is to get my log information to appear in Kibana's Discover tab.

There are a few good indications that the data successfully made it to Elasticsearch. For instance, when I run the curl 'http://es-host:9200/_cat/indices?v' command...

Capture4
Capture4.PNG656×525 69.4 KB

...the index successfully appears here. Kibana also sees this index in my production cluster, which seems like good news...

Capture3
Capture3.PNG1900×219 22.6 KB

...however, Kibana does not see the instance of Logstash in my production cluster, only the index (The number under "Logstash" remains 0):

Capture2
Capture2.PNG1926×671 35.9 KB

When I try to create an index pattern, the "logstash-2019.06.19-000001" index is completely missing from the list of indices I can usually choose to create an index pattern. I've let time pass for Kibana to update its UI but the problem persists. I am not accessing an Elasticsearch instance remotely. For now, Filebeat, Logstash, and Elasticsearch are running on the same machine.

2

I am not going to address the issue of what appears when you try to create an index pattern since that is not a logstash issue.

What do see in the logstash logs when it starts up? There should be monitoring related messages....

3

I'm running the latest versions of ELK. My configuration is based on this page:
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html#advanced-pipeline

The Logstash log information is here. There are no unusual errors.

To test my pipeline, I ran the command,
curl -XGET '10.200.26.185:9200/logstash-2019.06.19-000001/_search?pretty&q=response=200'

The terminal returned the enriched sample log data as expected.

4

There is no license check message, which suggests to me that you do not have monitoring enabled. If you run with --log.level debug what value does it print for xpack.monitoring.enabled?

6

I'm not really sure how or what command to utilize "--log.level". I restarted Logstash with xpack.monitoring.enabled manually set. Unfortunately, the issue persists.

7

Add '--log.level debug' to the logstash command line.

8

I appended '--log.level debug' at the end of the command that starts Logstash.

I ran the command bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic --log.level debug

I received a very, very long list of [DEBUG] messages. I couldn't find one relevant to the xpack.monitoring.enabled value.

9

I can think of no explanation for it not being there. I get these amongst the xpack configuration setting logged.

[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
10

I apologize. You're right. It was difficult to find with the log being so large. I found:

[DEBUG] 2019-06-24 08:29:28.345 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2019-06-24 08:29:28.345 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.collection.interval: 10000000000
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: 600000000000
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2019-06-24 08:29:28.347 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
11

OK, so you do not have monitoring enabled. What do you get for

grep xpack.monitoring.enabled /etc/logstash/logstash.yml

The markdown I used was three backticks on a line before and after the text. Thus

```
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
```

will appear as

[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"

If you precede a line with 4 spaces

    [2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
    [2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
    [2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
    [2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
    [2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"

Then it is rendered as

[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2019-06-24T15:39:30,746][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2019-06-24T15:39:30,747][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"

The effect of these on single lines is slightly different.

12

I had manually set the xpack.monitoring.enabled to true before running the debug. It's enabled, according to grep, but the debug while it runs says it's not.

[root@localhost logstash]# grep xpack.monitoring.enabled /etc/logstash/logstash.yml 
xpack.monitoring.enabled: true
13

If you have updated logstash.yml and logstash is not reading it then path.setting does not have the value you expect. The debug output should show path.settings a couple of lines above the xpack values. Something like

[2019-06-24T15:39:30,742][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
14

Yes, I see what you're talking about. Here is what the debug lists:

[DEBUG] 2019-06-24 11:09:18.834 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"

In fact, this directory does not exist for me (nor has it ever).

[root@localhost logstash]# pwd
/usr/share/logstash
[root@localhost logstash]# ls
bin           Gemfile       LICENSE.txt               modules     vendor
CONTRIBUTORS  Gemfile.lock  logstash-core             NOTICE.TXT  x-pack
data          lib           logstash-core-plugin-api  tools
[root@localhost logstash]# cd config
bash: cd: config: No such file or directory

I installed this using rpm. Should I change path.settings to "/etc/logstash" as that is the directory of my logstash.yml file? If so, how can I do this?

15

Okay. Progress is being made. To change path.settings temporarily, I appended
--path.settings /etc/logstash to the Logstash command line. The debug output showed me this:

[2019-06-24T11:40:57,743][DEBUG][logstash.runner          ] *xpack.monitoring.enabled: true (default: false)

The default output logs now show the license check message you were referring to. I'm not sure what to do at this point.

16

If you are getting this message by logstash still does not show up in kibana I would start a new thread saying that and asking for suggestions. I do not run xpack, so I have no idea.

17

I got it!

Thanks to this thread, I found out that the commented path for xpack.monitoring.elasticsearch.hosts began with "https://", when it was supposed to be "http://" (remove the 's'). This was why Logstash was unable to retrieve my license information and failed to fetch my X-pack information from elasticsearch.

My problem was fixed by first correcting the path.settings field so Logstash would not ignore my configurations, then correcting the xpack.monitoring.elasticsearch.hosts field. Now Logstash is being monitored like I had hoped!

image
image.png1855×336 19.5 KB

image
image.png1872×747 56.3 KB

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.