Logstash elasticsearch filter plugin authentication issues

[Dede_Pessu]

Hey Everyone,

I am currently trying to use the elasticsearch filter plugin for logstash to do some calculations/aggregations and it seems to be having issues connecting to our elasticsearch node.

based on the documentation here for Elasticsearch filter plugin for 6.8 https://www.elastic.co/guide/en/logstash/6.8/plugins-filters-elasticsearch.html it indicates that the only authentication configuration settings areuser, password and ca_file which i have passed the correct information for all parameters in my configuration below

 elasticsearch {
        index => "<index>"
        user                         => "<elastic_user>"
        password                     => "<elastic_password>"
        hosts                        => ["https://<elasticsearch_node_ip>:9200"]
        ca_file                       => '/path/to/ca_cert/elk_ca.cert.pem'
        query => "metric_type:metric AND file_name:%{[file_name]}"
        fields => {
          "pickup_time" => "pickup_time"
        }
      }

But i am getting the following error when logstash attemtps to start up with the above pipeline cofigurations

[2020-11-30T21:31:07,918][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"input_json_mt_metrics", :exception=>#<Manticore::ClientProtocolException: Received fatal alert:           bad_certificate>

Which is strange because i am referencing the same ca_cert in the elasticsearch output plugin which works fine

output {
  elasticsearch {
    index                        => "<index>"
    user                         => "<elastic_user>"
    password                     => "<elastic_password>"
    hosts                        => ["https://<elasticsearch_node_ip>:9200"]
    ssl                          => true
    ssl_certificate_verification => true
    cacert                       => '/path/to/ca_cert/elk_ca.cert.pem'
    keystore                     => '/path/to/keyfile/logstash.p12'
    keystore_password            => "<elastic_password>"
  }
}

So Im not sure why it the ca file is accepted in the elasticserach output plugin but not the elasticsearch filter plugin.

On my elasticnode these are the security setting in my elasticsearch.yml file

 xpack.security.enabled: true
 xpack.security.fips_mode.enabled: true
 xpack.security.authc.accept_default_password: false
 xpack.security.transport.ssl.enabled: true
 xpack.security.transport.ssl.verification_mode: certificate
 xpack.security.transport.ssl.key: /path/to/key/file.key.rsa.pem
 xpack.security.transport.ssl.certificate: /path/to/cert/file.combined.pem
 xpack.security.transport.ssl.certificate_authorities: ["/path/to/ca_cert/elk_ca.cert.pem"]
 xpack.security.http.ssl.enabled: true
 xpack.security.http.ssl.verification_mode: certificate
 xpack.security.http.ssl.client_authentication: required
 xpack.security.http.ssl.key: /path/to/key/file.key.rsa.pem
 xpack.security.http.ssl.certificate: /path/to/cert/file.combined.pem
 xpack.security.http.ssl.certificate_authorities: ["/path/to/ca_cert/elk_ca.cert.pem"]
 xpack.security.authc.realms.native.type: native
 xpack.security.authc.realms.native.order: 0
 xpack.security.authc.realms.pki1.type: pki
 xpack.security.authc.realms.pki1.order: 1
 
 xpack.security.authc.password_hashing.algorithm: PBKDF2

any ideas?

[TheHunter1]

Hi @Dede_Pessu,
I don't know if this will help you but for me I had an issue of certficate once, and the problem was in the input pipeline of losgtash (using beats to send data to logstash), my problem was that the key of logtash should be in pkcs8 format.

Best regards

[Dede_Pessu]

So it turns out that ssl client authentication is not supported by the filters in 6.8.