Using images from elastic/helm-charts
elasticsearch version: 6.8.2
logstash version: 6.7.0
Ever since enabling TLS for all ELK components, I am seeing the following error in logstash pod logs:
[2020-03-14T04:05:50,120][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>400, :url=>"https://<k8s-es-service-name>:9200/_bulk"}
400 Bad Request... meaning the client (logstash) is making a bad request to the server (elasticsearch). But I couldn't figure out what the problem is.
elasticsearch.yml:
elasticsearch.yml: |
# rest.action.multi.allow_explicit_index: false # tried this but did nothing
xpack.monitoring.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
logstash.yml:
logstash.yml:
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: ["<k8s-es-service-name>"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: logstash
# xpack.monitoring.elasticsearch.ssl.ca: "/usr/share/logstash/pemCerts/elastic-certificate.pem"
# xpack.monitoring.elasticsearch.ssl.certificate_authority: /usr/share/logstash/certs/elastic-certificates.p12
xpack.monitoring.elasticsearch.ssl.truststore.path: "/usr/share/logstash/certs/elastic-certificates.p12"
xpack.monitoring.elasticsearch.ssl.truststore.password: ""
xpack.monitoring.elasticsearch.ssl.keystore.path: "/usr/share/logstash/certs/elastic-certificates.p12"
xpack.monitoring.elasticsearch.ssl.keystore.password: ""
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: false
xpack.monitoring.collection.interval: 60s
xpack.monitoring.collection.pipeline.details.enabled: true
for xpack.monitoring.elasticsearch.hosts
, I have tried most sensible variations like:
["https://<k8s-es-service-name>:9200"]
["<k8s-es-service-name>:9200"]
["<k8s-es-service-name>"]
logstash.conf:
logstash.conf: |
output {
elasticsearch {
hosts => ["<k8s-es-service-name>"]
ssl => true
ssl_certificate_verification => true
keystore => "/usr/share/logstash/certs/elastic-certificates.p12"
keystore_password => ''
truststore => "/usr/share/logstash/certs/elastic-certificates.p12"
truststore_password => ''
manage_template => false
user => logstash_system
password => password
index => "<some-index>"
for hosts
, I have tried most sensible variations like:
["https://<k8s-es-service-name>:9200"]
["<k8s-es-service-name>:9200"]
["<k8s-es-service-name>"]
Would really appreciate any help. Thanks.