Logstash monitoring with Metricbeat "logstash-xpack" module - data not collected and not sent to ES

Elastic Stack Logstash
1

Hi,
I'm running version 7.7 of Logstash and Metricbeat, and I'm trying to setup stack monitoring using the metricbeat xpack modules.
So far I've enabled the elasticsearch-xpack module and it works fine, ES and Kibana monitoring data is available in the Stack Monitoring view.
However, for some reason logstash is not being monitored by metricbeat.

I know there was a similar post about this a couple of months ago but it was closed without a final solution.

I have ran the following command:
metricbeat test modules

and everything returns ok for the logstash module for both node and node_stats metrics,
but in ES the .monitoring-logstash-* is not created and logstash is not display in Stack Monitoring.

I have tried disabling and enabling the module and restarting metricbeat multiple times but nothing seems to work, and the log files in /var/log/metricbeat/ don't show any errors as well, in fact there is not even any occurances of the word logstash in any of them.

What can I do about this?
Thanks.

2

Hi @Ilya_Geller,

Can you share your list of enabled Metricbeat modules, via metricbeat modules list?

Assuming, logstash or logstash-xpack is enabled, can you also share that config too?

Lastly, can you also share your metricbeat.yml config? Feel free to scrub any sensitive information.

3

Hi thanks for the reply,

metricbeat modules list
Enabled:
elasticsearch-xpack
logstash-xpack
system

Disabled:
...<All the rest>...

logstash-xpack.yml

- module: logstash
  metricsets:
    - node
    - node_stats
  period: 10s
  hosts: ["localhost:9600"]
  username: "<user>"
  password: "<password>"
  xpack.enabled: true

metricbeat.yml

output.elasticsearch:
  hosts: ["https://<myhost>:9200"]
  ssl.certificate_authorities: ["<path_to_ca_cert>"]
  protocol: "https"
  username: "<user>"
  password: "<password>"

Everything that's not mentioned above is default.

4

Thanks for that. Nothing stands out to be off there.

I'm assuming you've checked for any warnings/errors in the Metricbeat log file?

Assuming you did the above, let's see if debug logs will tell us anything more.

Can you try starting Metricbeat in debug mode (by appending -d "*" to the end of the command you use to start Metricbeat, such as metricbeat -d "*") for about 15 seconds, then stopping Metricbeat and sharing the entire contents of the debug log?

5

Ok so, I have tried running in debug mode, and looking at the logs a repeating error stands out:

DEBUG [publisher] pipeline/client.go:230 Pipeline client receives callback 'onDroppedOnPublish' for event:
...<EVENT>...
... Private:<nil> TimeSeries:true}

where <EVENT> seems to contain all the needed data extracted by the metricbeat module.
Maybe this tells you anything?

6

If you run in regular mode (lose the -d option), do you see messages like this in the Metricbeat server log file:

2020-12-08T10:48:48.304-0500	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":156,"time":{"ms":81}},"total":{"ticks":644,"time":{"ms":148},"value":644},"user":{"ticks":488,"time":{"ms":67}}},"info":{"ephemeral_id":"fe599f11-d5ad-40e6-943e-2f46c14721c6","uptime":{"ms":60047}},"memstats":{"gc_next":12526624,"memory_alloc":10234848,"memory_total":68773016,"rss":2220032},"runtime":{"goroutines":32}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":3,"batches":6,"duplicates":12,"total":15},"read":{"bytes":5982},"write":{"bytes":49807}},"pipeline":{"clients":2,"events":{"active":0,"published":15,"total":15},"queue":{"acked":15}}},"metricbeat":{"logstash":{"node":{"events":12,"success":12},"node_stats":{"events":3,"success":3}}},"system":{"load":{"1":8.5332,"15":17.4814,"5":23.2773,"norm":{"1":0.5333,"15":1.0926,"5":1.4548}}}}}}

If so, can you share them? They should appear every ~10s when new data is pushed to Elasticsearch.

7

So apparently there is some issue with the built in remote_monitoring_user permissions for the metricbeat-* index.

I have manually added the all permission for metricbeat-* for the user through Kibana, restarted metricbeat and all of the sudden the missing logstash index was created and now Logstash appears in Stack Monitoring.

I guess something was missing and it couldn't create new indeces but that is weird since the ES and kibana indeces were created nontheless.

I have also manually added the read_ilm cluster privilege since metricbeat was complaining it could not retrieve the ILM policy, and also noticed repeating ...failed to check for alias 'metricbeat-7.7.0'... action [indeces:admin/aliases/get] is unauthorized for user

Could you get someone on the Elastic team to look into that?
For me its not a big deal to leave it like that, but this is probably something that needs to be checked.

8

The -xpack stack product modules (such as logstash-xpack and elasticsearch-xpack) do not index into metricbeat-*, but rather into custom, stack monitoring indices named .monitoring-*.

If you are seeing an issue with metricbeat-* indices, it must be due to another Metricbeat module (which, for you, looks like system).

Do you need the system module? If not, try disabling that and reverting the permission changes.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.