Logstash monitoring help not seeing in Kibana

Elastic Stack Logstash
1

Hi,

I am new to ELK and I have been trying to follow the guides and so far I have been abled to install version 7.17 in our 2 kibana nodes and our 3 Elasticsearch nodes. If I go to Kibana and look under Cluster Overview I can see the my ES and Kibana nodes but not my Logstash nodes.

I have not install any beats or metricbeats on logasth or ES or kibana.
I haven't install/comfigure any pipeline.

Do I need to install Metricbeats to see my nodes in Kibana?
Do I need to write a pipeline to start seeing my nodes ?

I used curl from logstash to ES without issue using my logstash_system username and password/ But from ES to logstash I can't .

So far

My logstash.yml file has the following configuration:

node.name: logstash-01
path.data: /var/lib/logstash

config.debug: true
log.level: debug

X-Pack Monitoring

xpack.monitoring.enabled: false
xpack.monitoring.Elasticsearch.username: logstash_system
xpack.monitoring.Elasticsearch.password: password
xpack.monitoring.Elasticsearch.hosts: ["https://es-01:9200", "https://es-02:9200"]
xpack.monitoring.Elasticsearch.ssl.certificate_authority: [ "/path/to/ca.crt" ]
xpack.monitoring.Elasticsearch.ssl.keystore.path: logstash.jks
xpack.monitoring.Elasticsearch.ssl.keystore.password: password
xpack.monitoring.Elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.Elasticsearch.sniffing: false
#xpack.monitoring.collection.interval: 10s
#xpack.monitoring.collection.pipeline.details.enabled: true

X-Pack Management

xpack.management.enabled: false
#xpack.management.pipeline.id: ["main", "apache_logs"]
xpack.management.Elasticsearch.username: logstash_admin_user
xpack.management.Elasticsearch.password: password

xpack.management.Elasticsearch.hosts: ["https://es-01:9200", "https://es-02:9200"]
xpack.management.Elasticsearch.ssl.certificate_authority: [ "ca.crt" ]
xpack.management.Elasticsearch.ssl.keystore.path: logstash.jks
xpack.management.Elasticsearch.ssl.keystore.password: password
xpack.management.Elasticsearch.ssl.verification_mode: certificate
#xpack.management.Elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s

I didn't find anything in the logs of Elasticsearch that can help me but the only error I have in the logstash logs is this.

[ERROR][logstash.inputs.metrics ] Failed to create monitoring event {:message=>"For path: events. Map keys: [:reloads, :pipelines]", :error=>"LogStash::Instrument::MetricStore::MetricNotFound"}

Please help

2

Hi @Mary2022 Welcome to the community.

It looks like you want to use the legacy deprecated monitoring... I would recommend moving over to metricbeat if this is going to be a production system.

But for legacy.... If so per the docs shouldn't

xpack.monitoring.enabled: false

Be

xpack.monitoring.enabled: true

Also, with this legacy method you need to set some settings in Elasticsearch as well. That is why I would suggest using the metricbeat method.

Also Please format your code going forward format button. It will make it easier for all of us to understand what your current settings are vs what you're trying to accomplish,.

3

It was a typo it is to true.
Yesterday I checked and the .monitor_logstash index is not in kibana either. I did a query and I saw the ones for kibana and Elasticsearch.

I also installed metricbeat in one of my logstash per the ELK documentation . The data showed up under Beat in Kibana but the logstash node still no show up under the ELK cluster.

Do I need to create the index or a pipeline? I didn't have to do that for ES or Kibana.

I see the logs how the index is created and new snapshots taken but they don't show up in Kibana.

Any help will be appreciated.

xpack.monitoring.enabled: true
xpack.monitoring.Elasticsearch.username: logstash_system
xpack.monitoring.Elasticsearch.password: password
xpack.monitoring.Elasticsearch.hosts: ["https://es-01:9200", "https://es-02:9200", "https://es-03:9200"]
xpack.monitoring.Elasticsearch.ssl.certificate_authority: [ "/etc/pki/ca.crt" ]
xpack.monitoring.Elasticsearch.ssl.keystore.path: /etc/pki/logstash.jks
xpack.monitoring.Elasticsearch.ssl.keystore.password: password
xpack.monitoring.Elasticsearch.ssl.verification_mode: certificate
#xpack.monitoring.Elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 10s
xpack.monitoring.collection.pipeline.details.enabled: true

xpack.management.enabled: true
#xpack.management.pipeline.id: ["main", "apache_logs"]
xpack.management.Elasticsearch.username: logstash_admin_user
xpack.management.Elasticsearch.password: password
xpack.management.Elasticsearch.hosts: ["https://es-01:9200", "https://es-02:9200", "https://es-03:9200"]
xpack.management.Elasticsearch.ssl.certificate_authority: ["/etc/pki/ca.crt"]
xpack.management.Elasticsearch.ssl.keystore.path:  /etc/pki/logstash.jks
xpack.management.Elasticsearch.ssl.keystore.password: password
xpack.management.Elasticsearch.ssl.verification_mode: certificate
#xpack.management.Elasticsearch.sniffing: false
#xpack.management.logstash.poll_interval: 5s
4

Please share your metricbeat configuration. If it data showed up under beats That means That most likely you did not have a configure correctly.

did you enable the logstash module and then configured the logstash module and then ran setup and then started beats.

Please share your metricbeat.yml and the modules.d/logstash.yml

Plus, if you use the beat you need to turn off the self monitoring / deprecated monitoring in logstash.yml

5

I enabled metricbeats and the logstash-xpack module and the beat-xpack module.
I ran setup no error there
I ran metricbeat test modules and I saw the following error:

beat...
  state...
    error... ERROR HTTP error 404 in: 404 Not Found
state...
    error... ERROR HTTP error 404 in: 404 Not Found
  state...
    error... ERROR HTTP error 404 in: 404 Not Found
state...
    error... ERROR HTTP error 404 in: 404 Not Found

logstash...
  node...
     error... ERROR timeout waiting for an event
 node_stats...
     error... ERROR timeout waiting for an event.

below my files

metricbeats.yml

#============================  Config Reloading ===============================

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.period: 10s

  reload.enabled: true

#======================Elasticsearch Template settings#=========================*
setup.template.settings:
    index.number_of_shards: 1
   index.codec: best_compression


 #===============================Kibana==================================*
setup.kibana:

     host: ["https://kiba-01:5601","https://kiba-02:5601"]
   #The protocol: https was added by me. It was not in the file.
     protocol: "https"
  
#=======================Elasticsearch Output==================================
output.elasticsearch:
   hosts: ["https://es-01:9200","https://es-02:9200",https://es-03:9200"]
protocol: "https"
username: "username"
password: "password"

#=============================Processors===============================
processors:
 -add_host_metadata: ~

#=============================Logging======
logging.level=debug
#=============================X-Pack Monitoring==============================
monitoring.enabled: true

beat-xpack.yml

- module: beat
  period: 10s
  hosts:  ["https://es-01:9200","https://es-02:9200",https://es-03:9200"]
  username: "username"
  password: "password"

  xpack.enabled=true

logstash-xpack.yml


- module: logstash
metricsets:
  - node
  - node_stats
period: 10s
hosts: ["https://logstash-i01:9600","https://logstash-i02:9600"]
username: username
password: password
xpack.enabled: true
6

Hi @Mary2022

I see a couple issues I think so lets back up a bit and will help clarify a few things I hope.

I am not sure if they are simple cut and past issues but there are a number of syntax / indentation issues with your files.

1st lets focus on monitoring logstash

Make sure you comment all this in the logstash yml

# xpack.monitoring.enabled: true
# xpack.monitoring.Elasticsearch.username: logstash_system
# xpack.monitoring.Elasticsearch.password: password
# xpack.monitoring.Elasticsearch.hosts: ["https://es-01:9200", "https://es-02:9200", "https://es-03:9200"]
# xpack.monitoring.Elasticsearch.ssl.certificate_authority: [ "/etc/pki/ca.crt" ]
# xpack.monitoring.Elasticsearch.ssl.keystore.path: /etc/pki/logstash.jks
# xpack.monitoring.Elasticsearch.ssl.keystore.password: password
# xpack.monitoring.Elasticsearch.ssl.verification_mode: certificate
# xpack.monitoring.Elasticsearch.sniffing: false
# xpack.monitoring.collection.interval: 10s
# xpack.monitoring.collection.pipeline.details.enabled: true

2nd the metricbeat.yml file should look like this... you will probably need to put fix ssl setting under the setup.kibana and output.elasticsearch settings once you get it working

#============================  Config Reloading ===============================

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 10s
  reload.enabled: true

#======================Elasticsearch Template settings#=========================*
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression


 #===============================Kibana==================================*
setup.kibana:
  # FIXED
  host: ["https://kiba-01:5601","https://kiba-02:5601"]
  # FIX AFTER YOU GET WORKING
  ssl.verification_mode: "none"

  
#=======================Elasticsearch Output==================================
output.elasticsearch:
  hosts: ["https://es-01:9200","https://es-02:9200",https://es-03:9200"]
  protocol: "https"
  username: "username"
  password: "password"
  # FIX AFTER YOU GET WORKING
  ssl.verification_mode: "none"

#=============================Processors===============================
processors:
 -add_host_metadata: ~

#=============================Logging======
# You had this incorrect logging.level=debug
logging.level: debug
#=============================X-Pack Monitoring==============================
# This self enables monitoring of this beat so you don't need the beats-xpack
monitoring.enabled: true

3rd remove / disable the beat-xpack.yml you do not need it and it is miss-configured.

4th your logstsash-xpack.yml should look like this.
Note I temporarily put it the ssl.verification_mode: "none" once you get it working you will need to put in the ssl information about the logstash ssl cert

- module: logstash
  period: 10s
  hosts: ["https://logstash-i01:9600","https://logstash-i02:9600"]
  username: username
  password: password
  xpack.enabled: true
  # FIX AFTER YOU GET WORKING
  ssl.verification_mode: "none"

run setup and test again...

Let us know how it goes...

BIG NOTE you need / should to actually be monitoring Elasticsearch as well to see so also enable the elasticsearch-xpack module and get that running too....
Not sure if you did that or not...

- module: elasticsearch
  xpack.enabled: true
  period: 10s
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "sakdfjhsldfkjhasdf"
  ssl.verification_mode : "none"

Using this... This is what I have now...

Screen Shot 2022-05-04 at 8.15.49 AM
Screen Shot 2022-05-04 at 8.15.49 AM2910×1760 381 KB

7

I did the changes you suggested above and I still feel like I am missing something. I know I am close but not usre what I am missing.

Now I only get the following error when I run metricbeats test modules

logstash...
  node...
     error... ERROR timeout waiting for an event
 node_stats...
     error... ERROR timeout waiting for an event.

I only have one pipeline running as a test. Maybe I should use a very basic pipeline? Its like is waiting for the data but I don't know what else to do to generate the events.

My kibana doesn't have an index pattern yet and I haven't create the indixes because I am waiting to first have to monitoring part working.

I also did the Elasticsearch module in the logstash server.

8

I realized something

Logstash module ... you probably have not enabled authentication ... or did you? if not

- module: logstash
  period: 10s
  hosts: ["https://logstash-i01:9600","https://logstash-i02:9600"]
  # username: username
  # password: password
  xpack.enabled: true
  # FIX AFTER YOU GET WORKING
  ssl.verification_mode: "none"

That error looks like it can not connect to logstash at "https://logstash-i02:9600" do you really have ssl enabled for the logstash api endpoint? Did you configure these settings in the logstash.yml... if so please share your entire logstash.yml.

Perhaps it is really just hosts: ["http://logstash-i01:9600","http://logstash-i02:9600"]

Not sure what that means... you setup up the elasticsearch-xpack module in metricbeat?

If So... Did the cluster monitoring show up in kibana?

9

Sorry, I tried to say that I enabled the elasticsearc-xpack module in metricbeat in the logstash node.

10

I forgot to mention that my 3 Elasticsearch nodes and 2 kibana nodes don't have metricbeats install or configured. They still using the legacy method.

Can I keep testing my logstash node using metricbeats and leave the rest using legacy collectors for now?

11

Hmmmm I am not sure... me I would not mix and match (not sure that is a supported configuration).. .could be a cause of some of your issues... you can try it.

12

Thanks so much, it is working now. I had API configured but the api.enabled was not to set for true.

I have a couple of questions moving forward.

What are the settings/properties I need to fix to perform SSL verification?

ssl.verification_mode: "none"

Why it is suggested to enable the Elasticsearch-xpack module in the logstash node?

13

If you are talking about this?

What I really mean is that you need / should be monitoring elasticsearch as well as logstash, the elasticsearch monitoring does not have to be on that logstash server .... but I thought that was the only place you were monitoring from

then you said this...

so you are monitoring elasticsearch that was the point... you need to be monitoring elasticsearch as well as logstash, where you monitoring elasticsearch from is up to you.

See here

14

First of all thanks so much for all your help. Our cluster has been working great.

Just a couple of updates, information, and questions.

We don't have a separate cluster for monitoring.
We installed metricbeat in both of our logstash servers.
We also enabled the following modules in our logstash server.

  • elastic-xpack
  • logstash-xpack
  • kibana-xpack

We did it to have some redundancy in case one of our logstash nodes/servers go down for any reason.

question:
I created a pipeline and added the new pipeline id to our logstash.yml file.

xpack.management.pipeline.id: ["main", "apache_logs"]

After doing this we noticed our logstash stop showing up under the cluster dashboard.

Is that normal?

15

Apologies I am a bit confused.

xpack.management.pipeline.id: ["main", "apache_logs"]

That setting is for centralized pipeline management not monitoring so I am unclear of the topic.

"After doing this we noticed our logstash stop showing up under the cluster dashboard."

I am not sure if you mean

a) Disappeared from the centralized Pipeline management screen... No Not Normal... probably a configuration issues...

b) Disappeared from the Stack Monitoring... no not normal.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.