Can't make Logstash monitoring work

Elastic Stack Logstash
1

I can't get Logstash monitoring to work.

I have a (one node) monitoring cluster already set up to which a production Elasticsearch cluster is sending monitoring information and I want to send Logstash monitoring stuff to it as well.

If I set Logstash log level to debug I get this in the log:

2019-05-10T15:29:57,705][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"http://monitoring_cluster_node:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", :body=>"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"request [/_xpack/monitoring/_bulk] contains unrecognized parameters: [interval], [system_api_version], [system_id]\"},\"status\":400}"}

x-pack security is not enabled on monitoring_cluster_node.

Logstash config:

path.data: /var/lib/logstash
pipeline.workers: 4
pipeline.output.workers: 1
pipeline.batch.size: 125
pipeline.batch.delay: 5
pipeline.unsafe_shutdown: false
path.config: /etc/logstash/conf.d
config.reload.automatic: false
config.reload.interval: 60
queue.type: memory
queue.page_capacity: 250mb
queue.max_events: 0
queue.max_bytes: 1024mb
queue.checkpoint.acks: 1024
queue.checkpoint.writes: 1024
queue.checkpoint.interval: 1000
dead_letter_queue.enable: false
log.level: debug
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: ["http://monitoring_cluster_node:9200"]

Logstash and Elasticsearch info:

[logstashnode]# /usr/share/logstash/bin/logstash --version
5.6.15
[logstashnode]# /usr/share/logstash/bin/logstash-plugin list | grep -i x-pack
x-pack
[logstashnode]# curl http://monitoring_cluster_node:9200
{
  "name" : "monitoring_cluster_node",
  "cluster_name" : "test_monitoring",
  "cluster_uuid" : "YpsioFsqRT6Hbid9QXFFFw",
  "version" : {
    "number" : "5.6.14",
    "build_hash" : "f310fe9",
    "build_date" : "2018-12-05T21:20:16.416Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.1"
  },
  "tagline" : "You Know, for Search"
}
2

I've got it working. I noticed that X-Pack Settings in Logstash | Logstash Reference [5.6] | Elastic says:

xpack.monitoring.elasticsearch.url

The Elasticsearch instances that you want to ship your Logstash metrics to. This might be the same Elasticsearch instance specified in the outputs section in your Logstash configuration, or a different one. This is not the URL of your dedicated monitoring cluster. Even if you are using a dedicated monitoring cluster, the Logstash metrics must be routed through your production cluster. You can specify a single host as a string, or specify multiple hosts as an array. Defaults to http://localhost:9200 .

So I changed logstash.yml settings to send to a node on the production cluster and now I have a Logstash node listed in the Monitoring section of Kibana on the monitoring cluster. But…

Monitoring Logstash | X-Pack for the Elastic Stack [5.6] | Elastic says:

To visualize Logstash as part of the Stack (shown below in Step 5), send metrics to your production cluster. Sending metrics to a dedicated monitoring cluster will show the Logstash metrics under the monitoring cluster.

Which I read as saying that metrics can be sent to the dedicated monitoring cluster as I was trying to do. The two pieces of documentation seem to offer conflicting advice about where to send metrics.