Kibana doesn't work with wildcards since Shield 2.1.0

Since I updated to ES 2.1.0, Shield 2.1.0 and Kibana 4.3.0, Kibana doesn't work properly.
If I set an index with a wildcard in the Settings tab (for example, logstash-*), when I go to the Discover tab, I get an error that says:

Fatal Error
  Courier Fetch: [index_not_found_exception] no such index, with: {"index":"[logstash-*]"}

  Version: 4.3.0
  Build: 9369

Error: [index_not_found_exception] no such index, with: {"index":"[logstash-*]"}
    at respond (http://X.X.X.X:5601/bundles/kibana.bundle.js:76160:16)
    at checkRespForFailure (http://X.X.X.X:5601/bundles/kibana.bundle.js:76123:8)
    at http://X.X.X.X:5601/bundles/kibana.bundle.js:74741:8
    at processQueue (http://X.X.X.X:5601/bundles/commons.bundle.js:42339:29)
    at http://X.X.X.X:5601/bundles/commons.bundle.js:42355:28
    at Scope.$eval (http://X.X.X.X:5601/bundles/commons.bundle.js:43583:29)
    at Scope.$digest (http://X.X.X.X:5601/bundles/commons.bundle.js:43394:32)
    at Scope.$apply (http://X.X.X.X:5601/bundles/commons.bundle.js:43691:25)
    at done (http://X.X.X.X:5601/bundles/commons.bundle.js:38140:48)
    at completeRequest (http://X.X.X.X:5601/bundles/commons.bundle.js:38338:8)

However, if I set in the Settings tab an index without wildcards (for example, logstash-type0-2015.11.25), I have no problems.
When I use a wildcard but Shield isn't installed, everything goes well, also.

Any idea?

1 Like

Anything in your ES or Shield access logs?

I have the same problem.

elasticsearch.yml

shield:
  authc:
    realms:
      esusers1:
        type: esusers
        order: 0
      ldap1:
        type: ldap
        order: 1
        url: "ldap://DC01.de:389"
        bind_dn: "BINDUSER"
        bind_password: 'changeit'
        user_search:
          base_dn: "OU=user,DC=domain,DC=de"
          attribute: mail
        group_search:
          base_dn: "OU=elk,OU=apps,DC=domain,DC=de"`

role_mapping.yml

  admin:
    - "CN=elk-admin,OU=elk,OU=apps,DC=domain,DC=de"
  marvel_user:
    - "CN=elk-kibana-marvel,OU=elk,OU=apps,DC=domain,DC=de"
  kibana4_team:
    - "CN=elk-kibana-team,OU=elk,OU=apps,DC=domain,DC=de"

roles.yml

  kibana4_team:
    cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
    indices:
      'team-*':
        privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
      '.kibana':
        privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

Elasticsearch Log:

[2015-11-27 09:53:40,919][INFO ][rest.suppressed          ] /team-*/_field_stats Params: {level=indices, index=team-*}
[[team-*]] IndexNotFoundException[no such index]
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.throwExceptionIfNoIndicesWereResolved(DefaultIndicesAndAliasesResolver.java:241)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.replaceWildcardsWithAuthorizedIndices(DefaultIndicesAndAliasesResolver.java:230)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.resolveIndicesAndAliases(DefaultIndicesAndAliasesResolver.java:81)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.resolve(DefaultIndicesAndAliasesResolver.java:74)
        at org.elasticsearch.shield.authz.InternalAuthorizationService.resolveIndices(InternalAuthorizationService.java:248)
        at org.elasticsearch.shield.authz.InternalAuthorizationService.authorize(InternalAuthorizationService.java:191)
        at org.elasticsearch.shield.action.ShieldActionFilter.apply(ShieldActionFilter.java:105)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:99)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77)
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
        at org.elasticsearch.client.support.AbstractClient.fieldStats(AbstractClient.java:792)
        at org.elasticsearch.rest.action.fieldstats.RestFieldStatsAction.handleRequest(RestFieldStatsAction.java:70)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:207)
        at org.elasticsearch.rest.RestController$RestHandlerFilter.process(RestController.java:281)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:262)
        at org.elasticsearch.shield.rest.ShieldRestFilter.process(ShieldRestFilter.java:77)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:265)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:348)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        ...

It seems like an issue with permissions because there isn't any problem when accessing with admin account.
The strange thing is my configuration is the same as the one I used for Shield 2.0.0 and as I read in the guide, there is no change in this part.

My roles.yml file contains the original data with the following part appended:

kibana4_type1_data:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    'logstash-type1-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    'logstash-type0-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

kibana4_type2_data:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    'logstash-type2-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    'logstash-type0-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

What I used to do with previous versions was configuring an index pattern such as logstash-* and user with roles kibana4_type1_data could access to logstash-type0-* and user with roles kibana4_type2_data could access to logstash-type1-* indices, while logstash-type0-* and logstash-type2-*. Now, it doesn't work. However, if I try to access to a, for example, index that follows the pattern logstash-type0-* with a user whose role is kibana4_type1_data, but it does work if I try to access to the specific index logstash-type0-2015.11.26.

Regarding to the logs Elasticsearch writes, they are exactly the same as the ones JoachimKunz posted with the single exception in the index name.

I have the same symptoms, using Found. I have a role for anonymous access:

# A user with readonly access to dashboards.
anonymous:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:admin/get
    '.kibana':
      - indices:admin/exists
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

This was working until some days ago when I upgraded ElasticSearch to 2.1.0, which supposedly upgraded to Shield 2.1.0 as well.

And yes, it seems to have to do with permissions. Admin accounts are working for me too...

The default role for Kibana users needs to have a permission added to allow the field stats api to be executed. Can you add indices:data/read/field_stats to the list of allowed privileges for the indices containing your data?

1 Like

Thanks a lot! This worked like a charm. The configuration that is working for me is:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
    '.kibana':
      - indices:admin/exists
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

Maybe this could be mentioned/fixed in the Shield docs, in the chapter related to Kibana...

Thanks a lot too! The configuration that is working for me is:

kibana4_team:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    '*':
      privileges: indices:data/read/field_stats
    'team-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get, indices:data/read/field_stats
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

Without a global access to "indices: data/read/field_stats" it doesn't work!

do you get the same error when only granting the privilege to the team-* pattern? And this is only on the discover page?

Hi all

It worked for me for one day. When I opened kibana today I got:

Discover: [index_not_found_exception] no such index, with: {"index":"[-*]"}

Have identical settings for anonymous just like you jgbarah and it doesn't work now !
that's really strange, my settings are:

elasticsearch.yml:

shield.authc.anonymous.authz_exception : false
shield.authc.anonymous.roles : anonymous
shield:
 authc:
   realms:
      esusers1:
        type: esusers
        order: 0

roles.yml:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
      - indices:data/write/bulk
    '.kibana':
      - indices:admin/exists
      - indices:data/write/bulk
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

Any ideas why anonymous access stopped working?

I had the exact same problem. Installing Kibana 4.3.1 resolved this issue for me.

Continuing the discussion from Kibana doesn't work with wildcards since Shield 2.1.0:

Yes I agree, after installing 4.3.1 problem dissapeared. Thanks!
However, I had to add some privileges to the annonymous user. Now my roles settings for anonymous looks like following:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
      - indices:data/write/bulk
      - indices:admin/create
      - indices:admin/template/get
    '.kibana':
      - indices:admin/template/get
      - indices:admin/exists
      - indices:data/write/bulk
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

I have come across with similar issue in the ES and Kibana 5.0 version, it seems like the problems on setting users and roles. But I have no ideas what is wrong. Can anyone help? Since it is another versio, thus I have opened a new post here.