Logstash does not appear in Elastic Monitoring - fleet provisioned agent

Elastic Stack Logstash
1

We have a fleet provisioned agent, installed it on our logstash server, however logstash does not appear in the monitoring dashboard.

We do seem to have logging and metrics data in:

logstash.stack_monitoring.node_stats
logstash.stack_monitoring.node
logstash.log

And some of the dashboards show metrics/logs, however they also show errors like:

Field logstash.host.name was not found.
Field logstash.pipeline.total.events.in was not found.
Fields logstash.pipeline.total.time.duration.ms, logstash.pipeline.total.events.out were not found.
Field logstash.pipeline.plugin.input.events.out was not found.
etc.

Any suggestions?

2

What Version Stack are you on?

Have you read this?

If you want to use the Elastic Agent integration for logstash ..

You should follow this...

1 Like
3

i had the same issue please read this it's some remarks i wrote about logstash (monitoring, security ...)

hope this helps

4

Yes, we followed the second one to the letter. We are on 8.15.1 @stephenb

5

Thanks @Abdarrahmane quite the detailed guide!
My problem is mainly that everything seems to be running, however the Logstash server does not appear in the Stack Monitoring dashboard. The agent is sending metrics, I see logs and metrics of the logstash server coming in, but I still can't monitor the server, dashboards are broken, etc

6

hello @neoaddix

regarding the logstash integration settings, please make sure that:

  • Make sure that Metrics (Stack Monitoring) is turned on, and Metrics (Technical Preview) is turned off, if you want to collect metrics from your Logstash instance

  • Under Metrics (Stack Monitoring), make sure the hosts setting points to your Logstash host URLs. By default, the integration collects Logstash monitoring metrics from localhost:9600. If that host and port number are not correct, update the hosts setting. If you configured Logstash to use encrypted communications, you must access it via HTTPS. For example, use a hosts setting like https://localhost:9600.

  • For the user: use a user with remote_monitoring_collector role

  • For the SSL configuration use : verification_mode: none (for test)

if all of this checks and you're still getting that problem then the solution currently in my humble opinion is monitoring using metricbeats
what i usually do is:
1- setup monitoring using the legacy way Monitoring Logstash (legacy) | Logstash Reference [8.15] | Elastic
write a pipeline and then go to stack monitoring to verify the appearance of my logstash node and monitor it

2-then switch to metricbeats monitoring, here is the guide to it :
(Collect Logstash monitoring data with Metricbeat | Logstash Reference [8.15] | Elastic)

here is an example of my conf files while monitoring with metricbeats:
logstash.yml

#xpack.monitoring.enabled: true #true if you are using legacy monitoring 
monitoring.enabled: false #flase if you are using metricbeat monitoring
xpack.monitoring.elasticsearch.username: logstash_sytem
xpack.monitoring.elasticsearch.password: password
xpack.monitoring.elasticsearch.hosts: ["https://x.x.x.x:9200", https://x.x.x.x:9200] #coordinating only nodes or data nodes 
xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/certs/elasticsearch-ca.pem
xpack.monitoring.elasticsearch.ssl.verification_mode: full
#xpack.monitoring.elasticsearch.ssl.verification_mode: none #use it to test comment .ssl.certificate and .ssl.key
xpack.monitoring.elasticsearch.ssl.certificate: /etc/logstash/certs/Logstash-1.crt
xpack.monitoring.elasticsearch.ssl.key: /etc/logstash/certs/Logstash-1.key

metricbeat.yml

# =================================== Kibana ===================================
setup.kibana:
  host: "https://x.x.x.x:5601"
# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # adresse_ip des nœuds de coordinations si vous en avez sinon les nœuds de data
  hosts: ["https://x.x.x.x:9200", "https://x.x.x.x:9200"]
  preset: balanced
  protocol: "https"
  username: "remote_monitoring_user"
  password: "password"
  ssl.certificate_authorities: /etc/metricbeat/elasticsearch-ca.pem

and /etc/metricbeat/logstash-xpack.yml

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["http://localhost:9600"]
  #username: "user"
  #password: "secret"
  metricsets:
      - node
      - node_stats
  period: 10s
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/metricbeat/elasticsearch-ca.pem"]

hope this helps

please keep us updated with the results!

7

Thanks @Abdarrahmane, your reply came in time to pull me out of the rabbit hole and switch from the agent to old fashioned xpack.monitoring. I do see Logstash in the Stack Monitoring screen now and it seems most metrics and logs are coming in too (if not all).

I do still notice that a lot of the preinstalled dashboards for logstash metrics are broken because they complain about fields missing. I still don't understand why though: are the dashboards wrong, is the data missing, are the fields wrongly named and what is the root cause of it

Maybe someone from the elastic team can give an answer, to me it feels there is a bug somewhere

I will keep this open for a bit to see if we can improve on the actual answer

1 Like