# .security index not found **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [August 24, 2016, 11:28pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873 "2016-08-24T23:28:45Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 24, 2016, 11:28pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/1 "2016-08-24T23:28:45Z")
Hi Team, Just now started with shield, Active Directory configurations and AD user authentication got failed. In the trace log i found .security index not found error, PFA snapshot. I created an esuser and it getting authenticated, not sure how to create .security index. Can you please suggest. Thanks & Regards ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [August 25, 2016, 10:33am UTC](https://discuss.elastic.co/t/security-index-not-found/58873/2 "2016-08-25T10:33:28Z")
Can you share the log message? In any case this is probably normal; the security index will not be created until you create a user or role via the API and you should not attempt to create it manually. ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 25, 2016, 6:41pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/3 "2016-08-25T18:41:29Z")
Hi Jay, PFB logs, let me know if you need more of it. Also attaching configurations i have done so far. gp-ops-es-admin is a AD group and i am expecting it will pull all users associated with that group. ![](https://us1.discourse-cdn.com/elastic/original/2X/8/88f9fe22ba8fed934a90d712703c11270f82a69b.JPG) \*test1 is an Active directory user and admin is an esuser \*remove duplicate lines from the log due to bytes restrictions Log: [2016-08-24 16:08:47,232][INFO][gateway] [es-node-1] recovered [72] indices into cluster\_state [2016-08-24 16:09:08,323][INFO][cluster.routing.allocation] [es-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[syslog-2016.08.19][0], [.kibana][0], [syslog-2016.08.19][0], [syslog-2016.08.19][3]] ...]). [2016-08-24 16:09:13,622][DEBUG][shield.authc.esnative] [es-node-1] user not found in cache, proceeding with normal authentication [2016-08-24 16:09:13,627][TRACE][shield.authc.esnative] [es-node-1] could not retrieve user [test1] because security index does not exist [.security] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:151) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2016-08-24 16:09:13,629][DEBUG][shield.authc.esusers] [es-node-1] user not found in cache, proceeding with normal authentication [2016-08-24 16:09:16,181][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist [2016-08-24 16:27:37,761][DEBUG][shield.authc.esnative] [es-node-1] user not found in cache, proceeding with normal authentication [2016-08-24 16:27:37,761][TRACE][shield.authc.esnative] [es-node-1] could not retrieve user [admin] because security index does not exist [.security] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:151) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:95) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteSingleIndex(IndexNameExpressionResolver.java:208) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2016-08-24 16:27:37,762][DEBUG][shield.authc.esusers] [es-node-1] user not found in cache, proceeding with normal authentication [2016-08-24 16:27:37,892][DEBUG][shield.authc.esusers] [es-node-1] authenticated user [admin], with roles [] [2016-08-24 16:27:46,196][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist [2016-08-24 20:05:04,561][INFO][node] [es-node-1] stopping ... [2016-08-24 20:05:04,567][INFO][watcher] [es-node-1] stopping watch service... [2016-08-24 20:05:05,009][INFO][watcher.watch] [es-node-1] stopped watch store [2016-08-24 20:05:05,009][INFO][watcher] [es-node-1] watch service has stopped [2016-08-24 20:05:05,552][INFO][node] [es-node-1] stopped [2016-08-24 20:05:05,553][INFO][node] [es-node-1] closing ... [2016-08-24 20:05:05,558][INFO][node] [es-node-1] closed ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [August 25, 2016, 8:17pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/4 "2016-08-25T20:17:09Z")
Do you have leading spaces in your configuration on the shield line? There should be messages about active directory but they aren't there ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 25, 2016, 9:55pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/5 "2016-08-25T21:55:28Z")
Yes, PFA better snapshot of it: I couldn't see any error in the log file and elasticsearch is also running fine. ![](https://us1.discourse-cdn.com/elastic/original/2X/5/55327f0618d0ff86f29d32603e35d1b8a8a610de.JPG) ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 25, 2016, 11:50pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/6 "2016-08-25T23:50:49Z")
Hello Jay, I tried with LDAP, and it actually tried to connect AD server but failed due to secure connection error. PFA snapshot. I can connect to AD server from ES instance but SSL is not enable, so i believe ES is trying to make a secure connection and that's why it's failing. Is there a way to make not secure connection with AD? Regards... ![](https://us1.discourse-cdn.com/elastic/original/2X/2/274fe82dc43c94d0a70d92c4faf4a628c3c3a243.JPG) ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [August 26, 2016, 9:56am UTC](https://discuss.elastic.co/t/security-index-not-found/58873/7 "2016-08-26T09:56:41Z")
Hi, Please remove the leading spaces before Shield. Also, rather than screenshots please copy the text in using a code block as it allows us to more easily provide suggestions. It should look like this (no leading space in front of shield). ``` shield: authc: realms: ``` ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 26, 2016, 4:27pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/8 "2016-08-26T16:27:59Z")
Hi Jay, It's yml file, if i removed space i will get error, PFB below: Please let me know how can i make no-secure connection between ES and Active directory? it looks like by default it is trying to make secure connection, which is failing. (my observation) Removed 2 space: Likely root cause: expected '', but found BlockMappingStart in 'reader', line 33, column 1: shield: ^ ``` at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225) at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53) at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:72) at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45) at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1071) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1058) at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88) at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) ``` Refer to the log for complete error details. [FAILED] Removed 1 space: Likely root cause: expected '', but found BlockMappingStart in 'reader', line 33, column 2: shield: ^ ``` at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225) at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158) at com.fasterxml.jackson.dataformat.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168) at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:342) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53) at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:72) at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45) at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46) at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1071) at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1058) at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88) at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:202) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:241) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) ``` Refer to the log for complete error details. [FAILED] Thanks & Regards ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 26, 2016, 4:38pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/9 "2016-08-26T16:38:11Z")
This is the current error i am getting, using LDAP: \*There is no SSL configured on AD server. ``` at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ``` Caused by: java.io.IOException: Unable to verify an attempt to to establish a secure connection to '--------:389' because an unexpected error was encountered during validation processing: javax.net.ssl.SSLPeerUnverifiedException: peer not authenticated at com.unboundid.ldap.sdk.LDAPConnectionInternals.(LDAPConnectionInternals.java:158) at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:859) ... 68 more [2016-08-26 09:36:12,537][DEBUG][shield.authc.ldap] [es-node-1] authentication failed for user [test1] java.io.IOException: failed to create a connection pool for realm [ldap1] as no LDAP servers are available at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:82) at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.findUserDN(LdapUserSearchSessionFactory.java:166) ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [August 26, 2016, 4:41pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/10 "2016-08-26T16:41:44Z")
The yml errors may be because of the `watcher:` line above that start with a space. Unless something should be nested, there should be no leading spaces. Also, it is trying to use ssl as you specified the URL with `ldaps`; use `ldap` for a plaintext connection. ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 26, 2016, 4:52pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/11 "2016-08-26T16:52:42Z")
Thanks Jay, i tried ldap and got below error: [2016-08-26 09:47:12,018][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist [2016-08-26 09:47:23,816][DEBUG][shield.authc.ldap] [es-node-1] user not found in cache, proceeding with normal authentication [2016-08-26 09:47:23,821][DEBUG][shield.authc.ldap] [es-node-1] unable to create connection pool for realm [ldap1] LDAPException(resultCode=49 (invalid credentials), errorMessage='80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580', diagnosticMessage='80090308: LdapErr: DSID-0C0903D0, comment: AcceptSecurityContext error, data 52e, v2580') at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2178) at com.unboundid.ldap.sdk.LDAPConnectionPool.createConnection(LDAPConnectionPool.java:1138) at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:1026) at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:913) at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:854) at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:801) at com.unboundid.ldap.sdk.LDAPConnectionPool.(LDAPConnectionPool.java:760) at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.createConnectionPool(LdapUserSearchSessionFactory.java:101) at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:79) ``` at java.lang.Thread.run(Thread.java:745) ``` [2016-08-26 09:47:23,821][DEBUG][shield.authc.ldap] [es-node-1] authentication failed for user [test1] java.io.IOException: failed to create a connection pool for realm [ldap1] as no LDAP servers are available at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.connectionPool(LdapUserSearchSessionFactory.java:82) at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.findUserDN(LdapUserSearchSessionFactory.java:166) at org.elasticsearch.shield.authc.ldap.LdapUserSearchSessionFactory.session(LdapUserSearchSessionFactory.java:140) at org.elasticsearch.shield.authc.ldap.support.AbstractLdapRealm.doAuthenticate(AbstractLdapRealm.java:50) at org.elasticsearch.shield.authc.support.CachingUsernamePasswordRealm.authenticate(CachingUsernamePasswordRealm.java:92) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2016-08-26 09:47:42,018][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist [2016-08-26 09:48:12,019][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist [2016-08-26 09:48:42,019][TRACE][shield.authc.esnative] [es-node-1] cannot poll for user changes since security index [.security] does not exist ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [August 26, 2016, 5:24pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/12 "2016-08-26T17:24:09Z")
I think active directory should be easier to setup. I noticed something else in your config formatting, `active_directory` needs to be indented beyond `realms` like: ``` shield: authc: realms: myad: type: active_directory url: ... ``` ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 26, 2016, 5:46pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/13 "2016-08-26T17:46:53Z")
Bingo, that worked. Thanks Jai. Now my user is getting authenticated and shield also pulled all AD realm but it's not mapped to any roles, PFB logs roles =[[]]. Logs: [2016-08-26 10:29:37,351][DEBUG][shield.authc.support] [es-node-1] the roles [[]], are mapped from the user [active\_directory] for realm [CN=test1,OU=users,OU=ops,OU=ccs,DC=,DC=,DC=,DC=com/active\_directory] [2016-08-26 10:29:37,355][DEBUG][shield.authc.activedirectory] [es-node-1] authenticated user [test1], with roles [[]] For role mapping i configured role\_mapping.yml file. I am assuming below should assign ES admin role to all AD users associated with group gp-ops-es-admin, in our case user "test1" role\_mapping.yml: admin: - "cn=gp-ops-es-admin,ou=ccs,dc=,dc=,dc=,dc=com" role\_mapping.yml file is in the CONF\_DIR. and not sure where to configure: shield.authc.active\_directory.files.role\_mapping ---
**Author:** ![piyush](https://avatars.discourse-cdn.com/v4/letter/p/ecb155/32.png) [@piyush](https://discuss.elastic.co/u/piyush)\ **Post date:** [August 26, 2016, 5:56pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/14 "2016-08-26T17:56:06Z")
It's resolved, Thank very much Jai, for your immediate assistance. configured: unmapped\_groups\_as\_roles: true ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:42pm UTC](https://discuss.elastic.co/t/security-index-not-found/58873/15 "2017-07-06T13:42:11Z")