.security index not found

security

(piyush) #1

Hi Team,
Just now started with shield, Active Directory configurations and AD user authentication got failed. In the trace log i found .security index not found error, PFA snapshot.

I created an esuser and it getting authenticated, not sure how to create .security index. Can you please suggest.

Thanks & Regards


(Jay Modi) #2

Can you share the log message? In any case this is probably normal; the security index will not be created until you create a user or role via the API and you should not attempt to create it manually.


(piyush) #3

Hi Jay,
PFB logs, let me know if you need more of it.

Also attaching configurations i have done so far. gp-ops-es-admin is a AD group and i am expecting it will pull all users associated with that group.

*test1 is an Active directory user and admin is an esuser
*remove duplicate lines from the log due to bytes restrictions

Log:
[2016-08-24 16:08:47,232][INFO ][gateway ] [es-node-1] recovered [72] indices into cluster_state
[2016-08-24 16:09:08,323][INFO ][cluster.routing.allocation] [es-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[syslog-2016.08.19][0], [.kibana][0], [syslog-2016.08.19][0], [syslog-2016.08.19][3]] ...]).
[2016-08-24 16:09:13,622][DEBUG][shield.authc.esnative ] [es-node-1] user not found in cache, proceeding with normal authentication
[2016-08-24 16:09:13,627][TRACE][shield.authc.esnative ] [es-node-1] could not retrieve user [test1] because security index does not exist
[.security] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:151)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-08-24 16:09:13,629][DEBUG][shield.authc.esusers ] [es-node-1] user not found in cache, proceeding with normal authentication
[2016-08-24 16:09:16,181][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist
[2016-08-24 16:27:37,761][DEBUG][shield.authc.esnative ] [es-node-1] user not found in cache, proceeding with normal authentication
[2016-08-24 16:27:37,761][TRACE][shield.authc.esnative ] [es-node-1] could not retrieve user [admin] because security index does not exist
[.security] IndexNotFoundException[no such index]
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:151)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:95)
at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:208)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-08-24 16:27:37,762][DEBUG][shield.authc.esusers ] [es-node-1] user not found in cache, proceeding with normal authentication
[2016-08-24 16:27:37,892][DEBUG][shield.authc.esusers ] [es-node-1] authenticated user [admin], with roles [[]]
[2016-08-24 16:27:46,196][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist
[2016-08-24 20:05:04,561][INFO ][node ] [es-node-1] stopping ...
[2016-08-24 20:05:04,567][INFO ][watcher ] [es-node-1] stopping watch service...
[2016-08-24 20:05:05,009][INFO ][watcher.watch ] [es-node-1] stopped watch store
[2016-08-24 20:05:05,009][INFO ][watcher ] [es-node-1] watch service has stopped
[2016-08-24 20:05:05,552][INFO ][node ] [es-node-1] stopped
[2016-08-24 20:05:05,553][INFO ][node ] [es-node-1] closing ...
[2016-08-24 20:05:05,558][INFO ][node ] [es-node-1] closed


(Jay Modi) #4

Do you have leading spaces in your configuration on the shield line? There should be messages about active directory but they aren't there


(piyush) #5

Yes, PFA better snapshot of it:

I couldn't see any error in the log file and elasticsearch is also running fine.


(piyush) #6

Hello Jay,
I tried with LDAP, and it actually tried to connect AD server but failed due to secure connection error. PFA snapshot.

I can connect to AD server from ES instance but SSL is not enable, so i believe ES is trying to make a secure connection and that's why it's failing.

Is there a way to make not secure connection with AD?

Regards...


(Jay Modi) #7

Hi,

Please remove the leading spaces before Shield. Also, rather than screenshots please copy the text in using a code block as it allows us to more easily provide suggestions. It should look like this (no leading space in front of shield).

shield:
    authc:
        realms:

(piyush) #8

Hi Jay,
It's yml file, if i removed space i will get error, PFB below:

Please let me know how can i make no-secure connection between ES and Active directory? it looks like by default it is trying to make secure connection, which is failing. (my observation)

Removed 2 space:
Likely root cause: expected '', but found BlockMappingStart
in 'reader', line 33, column 1:
shield:
^

    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)
    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:72)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
    at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1071)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1058)
    at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
    at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.
[FAILED]

Removed 1 space:
Likely root cause: expected '', but found BlockMappingStart
in 'reader', line 33, column 2:
shield:
^

    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)
    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
    at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
    at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342)
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:72)
    at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
    at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1071)
    at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1058)
    at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
    at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202)
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241)
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

Refer to the log for complete error details.
[FAILED]

Thanks & Regards


(piyush) #9

This is the current error i am getting, using LDAP:

*There is no SSL configured on AD server.

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Unable to verify an attempt to to establish a secure connection to '--------:389' because an unexpected error was encountered during validation processing: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated
at com.unboundid.ldap.sdk.LDAPConnectionInternals.(LDAPConnectionInternals.java:158)
at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:859)
... 68 more
[2016-08-26 09:36:12,537][DEBUG][shield.authc.ldap ] [es-node-1] authentication failed for user [test1]
java.io.IOException: failed to create a connection pool for realm [ldap1] as no LDAP servers are available
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:82)
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.findUserDN(LdapUserSearchSessionFactory.java:166)


(Jay Modi) #10

The yml errors may be because of the watcher: line above that start with a space. Unless something should be nested, there should be no leading spaces.

Also, it is trying to use ssl as you specified the URL with ldaps; use ldap for a plaintext connection.


(piyush) #11

Thanks Jay, i tried ldap and got below error:

[2016-08-26 09:47:12,018][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist
[2016-08-26 09:47:23,816][DEBUG][shield.authc.ldap ] [es-node-1] user not found in cache, proceeding with normal authentication
[2016-08-26 09:47:23,821][DEBUG][shield.authc.ldap ] [es-node-1] unable to create connection pool for realm [ldap1]
LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580')
at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2178)
at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:1138)
at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:1026)
at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:913)
at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:854)
at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:801)
at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:760)
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.createConnectionPool(LdapUserSearchSessionFactory.java:101)
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:79)

    at java.lang.Thread.run(Thread.java:745)

[2016-08-26 09:47:23,821][DEBUG][shield.authc.ldap ] [es-node-1] authentication failed for user [test1]
java.io.IOException: failed to create a connection pool for realm [ldap1] as no LDAP servers are available
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:82)
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.findUserDN(LdapUserSearchSessionFactory.java:166)
at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.session(LdapUserSearchSessionFactory.java:140)
at org.elasticsearch.shield.authc.ldap.support.AbstractLdapRealm.doAuthenticate(AbstractLdapRealm.java:50)
at org.elasticsearch.shield.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:92)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2016-08-26 09:47:42,018][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist
[2016-08-26 09:48:12,019][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist
[2016-08-26 09:48:42,019][TRACE][shield.authc.esnative ] [es-node-1] cannot poll for user changes since security index [.security] does not exist


(Jay Modi) #12

I think active directory should be easier to setup. I noticed something else in your config formatting, active_directory needs to be indented beyond realms like:

shield:
    authc:
        realms:
            myad:
                type: active_directory
                url: ...

(piyush) #13

Bingo, that worked. Thanks Jai.
Now my user is getting authenticated and shield also pulled all AD realm but it's not mapped to any roles, PFB logs roles =[[]].

Logs:
[2016-08-26 10:29:37,351][DEBUG][shield.authc.support ] [es-node-1] the roles [[]], are mapped from the user [active_directory] for realm [CN=test1,OU=users,OU=ops,OU=ccs,DC=,DC=,DC=,DC=com/active_directory]
[2016-08-26 10:29:37,355][DEBUG][shield.authc.activedirectory] [es-node-1] authenticated user [test1], with roles [[]]

For role mapping i configured role_mapping.yml file. I am assuming below should assign ES admin role to all AD users associated with group gp-ops-es-admin, in our case user "test1"

role_mapping.yml:
admin:

  • "cn=gp-ops-es-admin,ou=ccs,dc=,dc=,dc=,dc=com"

role_mapping.yml file is in the CONF_DIR. and not sure where to configure: shield.authc.active_directory.files.role_mapping


(piyush) #14

It's resolved, Thank very much Jai, for your immediate assistance.

configured:
unmapped_groups_as_roles: true


(system) #15