Tim Vernum:
This role mapping file has S=example.state , but your logs (and previous posts) show ST=example.state
Can you double check exactly what you've put there?
Yep. That's the problem.
For what it's worth, although I did notice the 'S' versus 'ST' difference Tim points out above, I never considered that it might be the cause of the authorization failure. The reason why may help others to avoid similar mistakes and assumptions. It also may help those who document ES security.
So here goes:
When I used the Java keytool to generate the keystore and truststore and ultimately retrieve the pem formatted public cert and private key (for use in the curl query), I read the following description in the keytool documentation (under the section titled "X.500 Distinguished Names", at the link keytool-Key and Certificate Management Tool)
When supplying a distinguished name string as the value of a `-dname` option, as for the `-genkeypair` command, the string must be in the following format:
CN=cName, OU=orgUnit , O=org , L=city , S=state , C=countryCode
A sample distinguished name string is
CN=Mark Smith, OU=Java, O=Oracle, L=Cupertino, S=California, C=US
and a sample command using such a string is
keytool -genkeypair -dname "CN=Mark Smith, OU=Java, O=Oracle, L=Cupertino, S=California, C=US" -alias mark
Note the 'S=California' entry in the example above. It was because of the above document that when I generated the X.509 PKI key pair for my example, I used 'S=example.state' instead of 'ST=example.state'.
But what's really confusing/misleading (at least, for me) is that even though 'S=example.state' was used in the 'keytool -genkeypair' command, I hadn't noticed (at least, not until Tim pointed it out above) that something in the process automatically changed the contents of the certificate to use 'ST' instead of 'S' in the stateName component.
Specifically, even though 'S=example.state' is specified in the -dname option of keytool, the command,
openssl x509 -in ./elasticsearch-2.4.6/config/shield/elasticsearch.pem -text
produces the following output:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1710276397 (0x65f0bf2d)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=example.state, L=example.city, O=example.org, OU=example.org.unit, CN=nosqlesuser
Validity
Not Before: Feb 25 16:45:13 2019 GMT
Not After : Nov 3 16:45:13 2023 GMT
Subject: C=US, ST=example.state, L=example.city, O=example.org, OU=example.org.unit, CN=nosqlesuser
Subject Public Key Info:
.....
Notice that the Issuer (the signer) and the Subject both specify the exact same name/value components as the distinguished name input to keytool; except that instead of 'S=example.state', they both specify 'ST=example.state'. This was quite a surprise to me. I'm not sure why keytool uses 'S=' instead of 'ST=', and I'm not sure why/how the 'S=' is replaced with 'ST=' in the certificates.
So the mistake I made was that I assumed that the DN I used when generating the key pair was what should be specified in the role_mapping.yml. But as Tim points out, it appears that it's the Subject that should be specified in role_mapping.yml; otherwise the desired role is not mapped correctly, and authorization will fail.
What made matters worse (for me at least), and what others should be aware of, is the fact that authentication worked correctly. That is, the intended username was correctly retrieved and successfully authenticated; whereas only authorization failed (because, of course, the intended role with the desired privileges could not be mapped to that username, for the reasons just described). I was assuming that the successful authentication meant that the certs must be okay, and authorization must be failing for a reason unrelated to those certs.
The last thing that sent me down the wrong logic path, and only added to my stupidity, was something in the Elasticsearch documentation itself. In the section titled "Mapping Users and Groups to Roles" in both the Shield and X-Pack documentation (see Mapping Users and Groups to Roles | Shield [2.4] | Elastic or Mapping Users and Groups to Roles | X-Pack for the Elastic Stack [5.4] | Elastic),
the following is stated:
PKI realms only support mapping users to roles, as there is no notion of a group in PKI. For example:
monitoring:
- "cn=Admin,ou=example,o=com"
user:
- "cn=John Doe,ou=example,o=com"
For what it's worth, when I read the above doc, I made another faulty assumption. I assumed that the fact that the locality, the state and the country components of the distinguished name were not included in the mapping specification above, they were probably irrelevant/ignored during the actual role mapping process. But of course, that's wrong.
If you specify a mapping like that shown in the Elasticsearch documentation, then the components and corresponding values in the distinguished name you use in the key pair(s) you generate, better match what's specified in the role_mapping.yml file. For the specific example above, the DN must consist of only the 'CN', 'OU', and 'O' component (names are case insensitive). Fortunately, the 'CN', 'OU', and 'O' components don't suffer from the issue described above that affects the 'S/ST' component.
So the moral of the story is, you not only have to make sure that your role_mapping.yml doesn't have any syntax or formatting errors (required dash, tabs vs spaces, etc.), but you must make sure that the entry in role_mapping.yml for which you wish to grant privileges, exactly matches all the corresponding name/value components of the Subject in the certificate generated for the associated user. It's not enough to use the value you supply in the -dname option of keytool.
Anyway, I want to thank the Elasticsearch team for their help and their patience with this issue. I apologize for taking us down the wrong path sometimes. But it was all very helpful and instructive; at least for me.
Thanks!
Brian