Logstash xpack.monitoring error for endpoint _bulk with TLS/SSL

[danksim (Daniel Sim)]

Using images from elastic/helm-charts
elasticsearch version: 6.8.2
logstash version: 6.7.0

---

Ever since enabling TLS for all ELK components, I am seeing the following error in logstash pod logs:

[2020-03-14T04:05:50,120][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>400, :url=>"https://<k8s-es-service-name>:9200/_bulk"}

400 Bad Request... meaning the client (logstash) is making a bad request to the server (elasticsearch). But I couldn't figure out what the problem is.

---

elasticsearch.yml:

elasticsearch.yml: |
  # rest.action.multi.allow_explicit_index: false # tried this but did nothing
  xpack.monitoring.enabled: true
  xpack.security.enabled: true
  xpack.security.transport.ssl.enabled: true
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
  xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
  xpack.security.http.ssl.enabled: true
  xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
  xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12

---

logstash.yml:

logstash.yml:
  xpack.monitoring.enabled: true
  xpack.monitoring.elasticsearch.hosts: ["<k8s-es-service-name>"]
  xpack.monitoring.elasticsearch.username: logstash_system
  xpack.monitoring.elasticsearch.password: logstash
  # xpack.monitoring.elasticsearch.ssl.ca: "/usr/share/logstash/pemCerts/elastic-certificate.pem"    
  # xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/certs/elastic-certificates.p12
  xpack.monitoring.elasticsearch.ssl.truststore.path: "/usr/share/logstash/certs/elastic-certificates.p12"
  xpack.monitoring.elasticsearch.ssl.truststore.password: ""
  xpack.monitoring.elasticsearch.ssl.keystore.path: "/usr/share/logstash/certs/elastic-certificates.p12"
  xpack.monitoring.elasticsearch.ssl.keystore.password: ""
  xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
  xpack.monitoring.elasticsearch.sniffing: false
  xpack.monitoring.collection.interval: 60s
  xpack.monitoring.collection.pipeline.details.enabled: true

for xpack.monitoring.elasticsearch.hosts, I have tried most sensible variations like:

["https://<k8s-es-service-name>:9200"]
["<k8s-es-service-name>:9200"]
["<k8s-es-service-name>"]

---

logstash.conf:

logstash.conf: |
  output {
    elasticsearch {
      hosts => ["<k8s-es-service-name>"]
      ssl => true
      ssl_certificate_verification => true
      keystore => "/usr/share/logstash/certs/elastic-certificates.p12"
      keystore_password => ''
      truststore => "/usr/share/logstash/certs/elastic-certificates.p12"
      truststore_password => ''
      manage_template => false
      user => logstash_system
      password => password
      index => "<some-index>"  

for hosts, I have tried most sensible variations like:

["https://<k8s-es-service-name>:9200"]
["<k8s-es-service-name>:9200"]
["<k8s-es-service-name>"]

---

Would really appreciate any help. Thanks.

[ikakavas (Ioannis Kakavas)]

Can you share some logs from Elasticsearch and more logs from the logstash elasticsearch output plugin ?

[danksim (Daniel Sim)]

Hi Ioannis, thanks for the reply.

I resolved this issue by adding the following values to logstash.yml:

logstash.yml:
  http.host: 0.0.0.0
  http.port: 9600
  ...