Can't make Logstash monitoring work

I can't get Logstash monitoring to work.

I have a (one node) monitoring cluster already set up to which a production Elasticsearch cluster is sending monitoring information and I want to send Logstash monitoring stuff to it as well.

If I set Logstash log level to debug I get this in the log:

2019-05-10T15:29:57,705][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://monitoring_cluster_node:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]\"},\"status\":400}"}

x-pack security is not enabled on monitoring_cluster_node.

Logstash config: /var/lib/logstash
pipeline.workers: 4
pipeline.output.workers: 1
pipeline.batch.size: 125
pipeline.batch.delay: 5
pipeline.unsafe_shutdown: false
path.config: /etc/logstash/conf.d
config.reload.automatic: false
config.reload.interval: 60
queue.type: memory
queue.page_capacity: 250mb
queue.max_events: 0
queue.max_bytes: 1024mb
queue.checkpoint.acks: 1024
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000
dead_letter_queue.enable: false
log.level: debug
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: ["http://monitoring_cluster_node:9200"]

Logstash and Elasticsearch info:

[logstashnode]# /usr/share/logstash/bin/logstash --version
[logstashnode]# /usr/share/logstash/bin/logstash-plugin list | grep -i x-pack
[logstashnode]# curl http://monitoring_cluster_node:9200
  "name" : "monitoring_cluster_node",
  "cluster_name" : "test_monitoring",
  "cluster_uuid" : "YpsioFsqRT6Hbid9QXFFFw",
  "version" : {
    "number" : "5.6.14",
    "build_hash" : "f310fe9",
    "build_date" : "2018-12-05T21:20:16.416Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  "tagline" : "You Know, for Search"

I've got it working. I noticed that X-Pack Settings in Logstash | Logstash Reference [5.6] | Elastic says:


The Elasticsearch instances that you want to ship your Logstash metrics to. This might be the same Elasticsearch instance specified in the outputs section in your Logstash configuration, or a different one. This is not the URL of your dedicated monitoring cluster. Even if you are using a dedicated monitoring cluster, the Logstash metrics must be routed through your production cluster. You can specify a single host as a string, or specify multiple hosts as an array. Defaults to http://localhost:9200 .

So I changed logstash.yml settings to send to a node on the production cluster and now I have a Logstash node listed in the Monitoring section of Kibana on the monitoring cluster. But…

Monitoring Logstash | X-Pack for the Elastic Stack [5.6] | Elastic says:

To visualize Logstash as part of the Stack (shown below in Step 5), send metrics to your production cluster. Sending metrics to a dedicated monitoring cluster will show the Logstash metrics under the monitoring cluster.

Which I read as saying that metrics can be sent to the dedicated monitoring cluster as I was trying to do. The two pieces of documentation seem to offer conflicting advice about where to send metrics.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.