"cannot poll for user changes since security index [.security] does not exist" prevents authorization

It does look like the role mappings are not picked up.

Just for clarity, there was a formatting bug in that version's output,

[2019-02-21 07:13:11,111][DEBUG][shield.authc.support ] [myHost] the roles [], are mapped from the user [pki] for realm [CN=myUser, OU=example.org.unit, O=example.org, L=example.city, ST=example.state, C=US/pki]

should be

[2019-02-21 07:13:11,111][DEBUG][shield.authc.support ] [myHost] the roles [], are mapped from the user [CN=myUser, OU=example.org.unit, O=example.org, L=example.city, ST=example.state, C=US] for realm [pki/pki1]

In later versions we have added debug output that contains the number of role mappings Elasticsearch could parse from the role_mapping.yml file so that would give us an indication as to whether the file is correctly parsed or not.

While we mull over possible causes, could you copy paste here the exact contents of

/scratch/es/elasticsearch/config/shield/role_mapping.yml

, you can use triple backquotes as follows

 ```
 your content here
 ```

so that the original formatting can be retained.

Also remove the myUser user from the file realm, because as Jay already mentioned

You can do this with

bin/shield/esusers/userdel myUser

Ioannis Kakavas wrote:

It does look like the role mappings are not picked up.

Agreed.

Just for clarity, there was a formatting bug in that version's output,

[2019-02-21 07:13:11,111][DEBUG][shield.authc.support ] [myHost] the roles [], are mapped from the user [pki] for realm [CN=myUser, OU=example.org.unit, O=example.org, L=example.city, ST=example.state, C=US/pki]

should be

[2019-02-21 07:13:11,111][DEBUG][shield.authc.support ] [myHost] the roles [

Ah, thanks for the clarification. I did notice that that particular log record seemed to have the user and realm swapped. So that was going to be one of my next questions.

While we mule over possible causes, could you copy paste here the exact contents of /scratch/es/elasticsearch/config/shield/role_mapping.yml?

Yes, the full contents of /scratch/es/elasticsearch/config/shield/role_mapping.yml are:

# Role mapping configuration file which has elasticsearch roles as keys
# that map to one or more user or group distinguished names

#roleA:   this is an elasticsearch role
#  - groupA-DN  this is a group distinguished name
#  - groupB-DN
#  - user1-DN   this is the full user distinguished name

#power_user:
#  - "cn=admins,dc=example,dc=com"
#user:
#  - "cn=users,dc=example,dc=com"
#  - "cn=admins,dc=example,dc=com"
#  - "cn=John Doe,cn=other users,dc=example,dc=com"

admin:
    - "CN=myUser,OU=example.org.unit,O=example.org,L=example.city,S=example.state,C=US"

For reference (and so that you don't have to scroll back up to one of the earlier postings), the contents of /scratch/es/elasticsearch/config/elasticsearch.yml are:

shield:
    enabled: true
    authc:
        realms:
            pki1:
                type: pki
                enabled: true
                files.role_mapping: /scratch/es/elasticsearch/config/shield/role_mapping.yml 
                order: 0 
            file1:
                type: file
                enabled: false
                order: 1 
    transport:
        ssl: true 
        ssl.client.auth: required
    http:
        ssl: true 
        ssl.client.auth: required
    ssl:
        keystore:
            path: /scratch/es/elasticsearch/config/shield/elasticsearch.keys
            password: myPswd
            key_password: myPswd
        truststore:
            path: /scratch/es/elasticsearch/config/shield/elasticsearch.trust
            password: myPswd

For what it's worth, I agree with your assessment that "role mappings are not picked up". I've done a bunch of trial-and-error with different configs to see if anything jumps out. I believe that I've verified that under the PKI realm, client authentication is working correctly; which I believe is an indication that the certs and passwords are consistent and being picked up. I just can't figure out why the admin role specified in role_mapping.yml is simply not being mapped to the CN=myUser in the DN.

Besides logging the number of role mappings as is done in later versions, is there any other logger that can be set (to DEBUG or TRACE) in the version I'm required to use that might show more about what is happening with role mapping?

What is the first version of ES that contains the log changes you describe above (the format change and logging the number of role mappings)?

Thanks,
Brian

Ioannis Kakavas wrote:

In later versions we have added debug output that contains the number of role mappings Elasticsearch could parse from the role_mapping.yml file so that would give us an indication as to whether the file is correctly parsed or not.

In an act of desperation, I installed Elasticsearch 5.6.15 and X-Pack 5.6.15, configured PKI in a fashion similar to what was done with ES/Shield 2.4.6 (same role_mapping.yml file), and re-ran the same curl query that was failing previously. I set the log level to debug because trace seemed to be too noisy to be helpful. The log output can be viewed at the following gist link:

The error returned by the curl command is identical to the error returned when executed against ES 2.4.6. And the log output shows a similar reason for the failure; that is,

the roles [[]], are mapped from the user [CN=myUser, OU=example.org.unit, O=example.org, L=example.city, ST=example.state, C=US] using file [role_mapping.yml] for realm [pki/pki1]

Of course, as Ioannis describes above, there is more log output and the formatting issue for the log record above has been corrected. But I still don't see anything that might explain why the admin role is failing to be mapped to myUser. There is this log record:

[INFO ][o.e.x.s.a.s.m.NativeRoleMappingStore] [myHost] The security index is not yet available - no role mappings can be loaded

But from prior responses to this thread, I'm assuming that the non-existent security index is not the cause of the issue, right?

Finally, before anyone asks, because X-Pack instead of Shield was used, the PKI properties in the elasticsearch.yml file were changed to the following values:

action.auto_create_index: ".security,.security*,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*"

xpack.security:
    enabled: true
    authc:
        realms:
            pki1:
                type: pki
                enabled: true
                order: 0
    transport:
        ssl: 
            enabled: true
            client_authentication: required
            keystore:
                path: /scratch/es/elasticsearch/config/x-pack/elasticsearch.keys
                password: No_Sql_00
                key_password: No_Sql_00
            truststore:
                path: /scratch/es/elasticsearch/config/x-pack/elasticsearch.trust
                password: No_Sql_00
    http:
        ssl:
            enabled: true
            client_authentication: required
            keystore:
                path: /scratch/es/elasticsearch/config/x-pack/elasticsearch.keys
                password: No_Sql_00
                key_password: No_Sql_00

Although I'm still required to use ES 2.4.6, I'm hoping that working with ES 5.6.15 will shed more light on what I'm doing wrong. If anyone has any ideas on what may be wrong, I'd greatly appreciate hearing those ideas or suggestions.

Thanks,
Brian

This role mapping file has S=example.state, but your logs (and previous posts) show ST=example.state

Can you double check exactly what you've put there?

Tim Vernum:

This role mapping file has S=example.state , but your logs (and previous posts) show ST=example.state

Can you double check exactly what you've put there?

Yep. That's the problem.

For what it's worth, although I did notice the 'S' versus 'ST' difference Tim points out above, I never considered that it might be the cause of the authorization failure. The reason why may help others to avoid similar mistakes and assumptions. It also may help those who document ES security.

So here goes:

When I used the Java keytool to generate the keystore and truststore and ultimately retrieve the pem formatted public cert and private key (for use in the curl query), I read the following description in the keytool documentation (under the section titled "X.500 Distinguished Names", at the link https://docs.oracle.com/javase/7/docs/technotes/tools/solaris/keytool.html)

When supplying a distinguished name string as the value of a  `-dname`  option, as for the  `-genkeypair`  command, the string must be in the following format:

CN=cName, OU=orgUnit , O=org , L=city , S=state , C=countryCode

A sample distinguished name string is

CN=Mark Smith, OU=Java, O=Oracle, L=Cupertino, S=California, C=US

and a sample command using such a string is

keytool -genkeypair -dname "CN=Mark Smith, OU=Java, O=Oracle, L=Cupertino, S=California, C=US" -alias mark

Note the 'S=California' entry in the example above. It was because of the above document that when I generated the X.509 PKI key pair for my example, I used 'S=example.state' instead of 'ST=example.state'.

But what's really confusing/misleading (at least, for me) is that even though 'S=example.state' was used in the 'keytool -genkeypair' command, I hadn't noticed (at least, not until Tim pointed it out above) that something in the process automatically changed the contents of the certificate to use 'ST' instead of 'S' in the stateName component.

Specifically, even though 'S=example.state' is specified in the -dname option of keytool, the command,

openssl x509 -in ./elasticsearch-2.4.6/config/shield/elasticsearch.pem -text

produces the following output:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1710276397 (0x65f0bf2d)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=example.state, L=example.city, O=example.org, OU=example.org.unit, CN=nosqlesuser
        Validity
            Not Before: Feb 25 16:45:13 2019 GMT
            Not After : Nov  3 16:45:13 2023 GMT
        Subject: C=US, ST=example.state, L=example.city, O=example.org, OU=example.org.unit, CN=nosqlesuser
        Subject Public Key Info:
.....

Notice that the Issuer (the signer) and the Subject both specify the exact same name/value components as the distinguished name input to keytool; except that instead of 'S=example.state', they both specify 'ST=example.state'. This was quite a surprise to me. I'm not sure why keytool uses 'S=' instead of 'ST=', and I'm not sure why/how the 'S=' is replaced with 'ST=' in the certificates.

So the mistake I made was that I assumed that the DN I used when generating the key pair was what should be specified in the role_mapping.yml. But as Tim points out, it appears that it's the Subject that should be specified in role_mapping.yml; otherwise the desired role is not mapped correctly, and authorization will fail.

What made matters worse (for me at least), and what others should be aware of, is the fact that authentication worked correctly. That is, the intended username was correctly retrieved and successfully authenticated; whereas only authorization failed (because, of course, the intended role with the desired privileges could not be mapped to that username, for the reasons just described). I was assuming that the successful authentication meant that the certs must be okay, and authorization must be failing for a reason unrelated to those certs.

The last thing that sent me down the wrong logic path, and only added to my stupidity, was something in the Elasticsearch documentation itself. In the section titled "Mapping Users and Groups to Roles" in both the Shield and X-Pack documentation (see https://www.elastic.co/guide/en/shield/current/mapping-roles.html or https://www.elastic.co/guide/en/x-pack/5.4/mapping-roles.html),
the following is stated:

PKI realms only support mapping users to roles, as there is no notion of a group in PKI. For example:

monitoring:
  - "cn=Admin,ou=example,o=com"
user:
  - "cn=John Doe,ou=example,o=com"

For what it's worth, when I read the above doc, I made another faulty assumption. I assumed that the fact that the locality, the state and the country components of the distinguished name were not included in the mapping specification above, they were probably irrelevant/ignored during the actual role mapping process. But of course, that's wrong.

If you specify a mapping like that shown in the Elasticsearch documentation, then the components and corresponding values in the distinguished name you use in the key pair(s) you generate, better match what's specified in the role_mapping.yml file. For the specific example above, the DN must consist of only the 'CN', 'OU', and 'O' component (names are case insensitive). Fortunately, the 'CN', 'OU', and 'O' components don't suffer from the issue described above that affects the 'S/ST' component.

So the moral of the story is, you not only have to make sure that your role_mapping.yml doesn't have any syntax or formatting errors (required dash, tabs vs spaces, etc.), but you must make sure that the entry in role_mapping.yml for which you wish to grant privileges, exactly matches all the corresponding name/value components of the Subject in the certificate generated for the associated user. It's not enough to use the value you supply in the -dname option of keytool.

Anyway, I want to thank the Elasticsearch team for their help and their patience with this issue. I apologize for taking us down the wrong path sometimes. But it was all very helpful and instructive; at least for me.

Thanks!
Brian

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.