[X-Pack] Logstash PKI client authentication not working

[Bozzerreaneaux (Bozho Bekriev)]

Hello,

I am struggling with getting Logstash to present a client certificate to ES instance. After couple discussions and misunderstandings from my side I realized that X-Pack supports client certificates through the keystore options. I thought I had set up everything correctly but I am getting a "missing authentication token for REST request" when I start Logstash.

When I start logstash in the console with this command:
logstash.bat -f C:\Logstash\conf.d\logstash.json

I am getting:

[2017-09-25T13:06:00,663][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk] is unauthorized for user [app01]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk] is unauthorized for user [app01]\"},\"status\":403}"}
[2017-09-25T13:06:04,620][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://logstash_system:xxxxxx@ELK02.devops.test:9200/, :path=>"/"}

Then, In the stdout log for Logstash I see:

"missing authentication token for REST request"

The error makes me think that Logstash is not able to attach the client certificate to the ES calls.

Could I please ask someone to verify my configuration:

ES:

xpack:
    security:
        authc:
            realms:
                pki1:
                    type: pki
                    order: 0
                    certificate_authorities: "x-pack/certificate-bundle/ca/ca.crt"

LS:

 output {
  stdout { codec => rubydebug  } elasticsearch {
    hosts => [ "https://ELK02.devops.test:9200" ]
	ssl => true
	cacert => "C:/Program Files/filebeat-5.5.2/certs/ca.crt"
	keystore => "C:/Program Files/filebeat-5.5.2/certs/Logstash-client-app1.jks"
	keystore_password => "******"
    manage_template => false
    index => "app1-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

The client certificate was created with certgen tool and I put "app01" under instance during the creation so in the certificate CN=app01

The following is output from the keytool -list command

Alias name: 1
Creation date: Sep 15, 2017
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=app01
Issuer: CN=Elastic Certificate Tool Autogenerated CA

I have created the following role mapping using the mapping API:

PUT _xpack/security/role_mapping/Logstash_users
{
  "roles" : [ "app1_logstash" ],
  "rules" : { "field" : {
    "dn" : "cn=app01"
  } },
  "enabled": true
}

[Bozzerreaneaux (Bozho Bekriev)]

It seems the role mapping was not picking the user set under the client cert DN.

I had set it before as "cn=app01". Once I recreated the mapping and changed this to"CN=app01", it worked.

So, it seems this field is case sensitive.