Logstash monitoring Encountered a retryable error. Will Retry with exponential backoff

[vaclav1]

Hello,

I have just swapped to use TLS on http and configured api_key as per Configuring Security in Logstash | Logstash Reference [7.9] | Elastic

My logstash.yml file:

# X-Pack Monitoring
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["https://xx", "https://xx", "https://xx"]
xpack.monitoring.collection.interval: 1m
xpack.monitoring.elasticsearch.api_key: 5CsC1nkB_0IBCcsS_wsH:PRXjXbYiTiO9kpvV5D2n4w
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.ssl.certificate_authority: /app/products/logstash/conf/certs/elasticsearch-ca.pem

After I start logstash I see logstash trying to send monitoring data to Elasticsearch but is not successful:

[2021-06-04T10:11:26,602][ERROR][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][07d47826dd904d67fbd9d3c000e0e7de6f4ff20dc08e227a3fb2994c36b4101a] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"https://10.194.144.63:9200/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s"}

403 means api_key has insufficient privileges?

I defined this way:

{
  "name": "logstash_monitoring_czf2", 
  "role_descriptors": {
    "logstash_monitoring": { 
      "cluster": ["monitor"],
      "index": [
        {
          "names": [".monitoring-ls-*"],
          "privileges": ["create_index", "create", "write"]
        }
      ]
    }
  }
}