PKI authentication in logstash Elastic Search Output not working

(Sushant Sood) #1

Hi all, I am trying to achieve PKI auth in Logstash elastic search output plugin using shield trail license .Below are my configs

Logstash :

output {
elasticsearch {
cluster => "abc"
host => ["host1","host2:port"]
protocol : "http"
ssl => true
keystore => "/etc/pki/logstashStr.jks"
keystore_password => "*****"
cacert => "etc/pki/servercert.crt"
index => "logstash-mm-dd-yyyy"

Still the exception is coming in logstash logs as :
AuthenticationException[missing Authentication token for REST service status 401]

My ELK stack is : Kibana -4.0.3
ES - 1.7.1
logstash - 1.5.3
LSF - 0.4.0

Please help in enabling PKI . I have done required configs in Shield for PKI also .

Logstash certificate not trusted
(Magnus Bäck) #2

What does your Shield realm configuration look like? Is there anything in the server-side logs about the authentication attempts?

index => "logstash-mm-dd-yyyy"

I think you mean this:

index => "logstash-%{+MM-dd-YYYY}"

(Sushant Sood) #3

Shield realm is very basic :

shield.authc.realms :
type: esusers
type: pki
order: 1

(Sushant Sood) #4

in Shield access logs i found out error where it mentioned Anonymous Access denied.

(Sushant Sood) #5

I get success only if i am providing user and password in logstash elasticsearch plugin but PKI auth is not working .

(Mike Simos) #6


Does certificate on logstash specify the username in the common name (CN)? And what you specified in the CN map to something in the role_mapping.yml?

(Sushant Sood) #7

CN of logstash certificate contains the logstash server name and in role_mapping.xml i have mapped power_user role to CN Map i.e CN=servername ,OU=CompanyUnit,O=Company.

Can you please elaborate if i need to create a user in Elasticsearch in esusers with that server name ?

(Jay Modi) #8

have you enabled SSL on your cluster? Can you share all of the shield settings?

(Mike Simos) #9

You should map the user to the logstash role. I don't think the power_user has sufficient privileges. See the following:

Also I'd check the log file on the Elasticsearch cluster for any error messages related to SSL.

(Sushant Sood) #10

These are my shield configs :

shield.ssl.keystore.path: /etc/pki/elasticWild.jks
shield.ssl.keystore.password: *****
shield.http.ssl: true
shield.transport.ssl: true

shield.audit.enabled: true
######################################Cipher Config########################
shield.ssl.ciphers: TLS_RSA_WITH_AES_128_CBC_SHA

type: esusers
order: 0

type: pki
order: 1

(Sushant Sood) #11

As per the error in logstash logs which says Authentication token missing in the request . it seems logstash is not sending the cert details to the elastic search and elastic search is not getting any PKI cert creds . Is it really an issue with shield config ?

(Jay Modi) #12

It looks like you are using the logstash HTTP output. In order for PKI authentication to work with http you will need to set:

shield.http.ssl.client.auth: optional

This setting will ask the clients to present a certificate for authentication, but it does not require the certificate so basic authentication will still work.

(Sushant Sood) #13

Hi I made the suggested changes in shield config , now the user is being authenticated but now i am getting Autharization Exception . Even though i have already defined the mapping in role_mapping.yml

[2015-10-26 15:27:06,435][DEBUG][shield.ssl ] no truststore defined. using keystore [/etc/pki/elasticWild.jks] as truststore
[2015-10-26 15:27:07,307][DEBUG][ ] [Sinister] the roles [[]], are mapped from these [pki] groups [[]] for realm [pki/pki1]
[2015-10-26 15:27:07,308][DEBUG][shield.audit.logfile ] [Sinister] [transport] [access_denied] origin_type=[rest], origin_address=[], principal=[***], action=[indices:data/write/bulk], request=[BulkRequest]
[2015-10-26 15:27:08,281][DEBUG][shield.ssl ] no truststore defined. using keystore [/etc/pki/elasticWild.jks] as truststore
[2015-10-26 15:27:09,331][DEBUG][ ] [Sinister] the roles [[]], are mapped from these [pki] groups [[]] for realm [pki/pki1]

(Sushant Sood) #14

shield.http.ssl.client.auth: optional , does this config mean that we are only authenticating the Server and bypassing the Client machine auth as part of SSL authentication ? Means we are not implementing the Mutual Authentication ?

(Sushant Sood) #15

For more details ,below is my details in role-mapping.yml


  • "CN=server name,OU=unitname,O=unit"

This value is matching with the CN,OU and O in the certificate and also with the principal coming in the logs

2015-10-26 15:27:07,308][DEBUG][shield.audit.logfile ] [Sinister] [transport] [access_denied] origin_type=[rest], origin_address=[], principal=[***], action=[indices:data/write/bulk], request=[BulkRequest]

(Jay Modi) #16

It appears as though there is some issue with the role mapping. When there is a role mapping that matches the user DN, you would see the following line:

the roles [admin], are mapped from the user [CN=server name,OU=unitname,O=unit] for realm [pki/pki1]

Is the formatting correct in the yml file? Only spaces and no tabs? Is that the full Subject DN in your certificate?

This means that mutual authentication is not required but is requested; if the client does not provide a certificate then they will have to authenticate using another method such as username/password. You can change optional to required to require all HTTP clients to perform mutual authentication.

(Sushant Sood) #17

I checked the role mapping file and i have only spaces and no tabs and i have added below entry :


and it matches with the certificate Subject DN .

(Sushant Sood) #18

1 more thing , can we add 2 Subject DNs in 1 role like below :


  • "CN=server1,OU=unitname,O=unit"
  • "CN=server2,OU=unitname,O=unit"

Although currently i am using only 1 subject DN.

(Jay Modi) #19

Yes, you can have multiple subject DNs like in your example. Were you able to get it working?

(Sushant Sood) #20

Hi Jay, I could achieve the PKI auth but Authorization still not working it is not able to get the role from the role-mapping.