Kibana doesn't work with wildcards since Shield 2.1.0

security

(Victor Cano) #1

Since I updated to ES 2.1.0, Shield 2.1.0 and Kibana 4.3.0, Kibana doesn't work properly.
If I set an index with a wildcard in the Settings tab (for example, logstash-*), when I go to the Discover tab, I get an error that says:

Fatal Error
  Courier Fetch: [index_not_found_exception] no such index, with: {"index":"[logstash-*]"}

  Version: 4.3.0
  Build: 9369

Error: [index_not_found_exception] no such index, with: {"index":"[logstash-*]"}
    at respond (http://X.X.X.X:5601/bundles/kibana.bundle.js:76160:16)
    at checkRespForFailure (http://X.X.X.X:5601/bundles/kibana.bundle.js:76123:8)
    at http://X.X.X.X:5601/bundles/kibana.bundle.js:74741:8
    at processQueue (http://X.X.X.X:5601/bundles/commons.bundle.js:42339:29)
    at http://X.X.X.X:5601/bundles/commons.bundle.js:42355:28
    at Scope.$eval (http://X.X.X.X:5601/bundles/commons.bundle.js:43583:29)
    at Scope.$digest (http://X.X.X.X:5601/bundles/commons.bundle.js:43394:32)
    at Scope.$apply (http://X.X.X.X:5601/bundles/commons.bundle.js:43691:25)
    at done (http://X.X.X.X:5601/bundles/commons.bundle.js:38140:48)
    at completeRequest (http://X.X.X.X:5601/bundles/commons.bundle.js:38338:8)

However, if I set in the Settings tab an index without wildcards (for example, logstash-type0-2015.11.25), I have no problems.
When I use a wildcard but Shield isn't installed, everything goes well, also.

Any idea?


(Mark Walkom) #2

Anything in your ES or Shield access logs?


(Joachim Kunz) #3

I have the same problem.

elasticsearch.yml

shield:
  authc:
    realms:
      esusers1:
        type: esusers
        order: 0
      ldap1:
        type: ldap
        order: 1
        url: "ldap://DC01.de:389"
        bind_dn: "BINDUSER"
        bind_password: 'changeit'
        user_search:
          base_dn: "OU=user,DC=domain,DC=de"
          attribute: mail
        group_search:
          base_dn: "OU=elk,OU=apps,DC=domain,DC=de"`

role_mapping.yml

  admin:
    - "CN=elk-admin,OU=elk,OU=apps,DC=domain,DC=de"
  marvel_user:
    - "CN=elk-kibana-marvel,OU=elk,OU=apps,DC=domain,DC=de"
  kibana4_team:
    - "CN=elk-kibana-team,OU=elk,OU=apps,DC=domain,DC=de"

roles.yml

  kibana4_team:
    cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
    indices:
      'team-*':
        privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
      '.kibana':
        privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

(Joachim Kunz) #4

Elasticsearch Log:

[2015-11-27 09:53:40,919][INFO ][rest.suppressed          ] /team-*/_field_stats Params: {level=indices, index=team-*}
[[team-*]] IndexNotFoundException[no such index]
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.throwExceptionIfNoIndicesWereResolved(DefaultIndicesAndAliasesResolver.java:241)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.replaceWildcardsWithAuthorizedIndices(DefaultIndicesAndAliasesResolver.java:230)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.resolveIndicesAndAliases(DefaultIndicesAndAliasesResolver.java:81)
        at org.elasticsearch.shield.authz.indicesresolver.DefaultIndicesAndAliasesResolver.resolve(DefaultIndicesAndAliasesResolver.java:74)
        at org.elasticsearch.shield.authz.InternalAuthorizationService.resolveIndices(InternalAuthorizationService.java:248)
        at org.elasticsearch.shield.authz.InternalAuthorizationService.authorize(InternalAuthorizationService.java:191)
        at org.elasticsearch.shield.action.ShieldActionFilter.apply(ShieldActionFilter.java:105)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:99)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77)
        at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
        at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:52)
        at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient.doExecute(BaseRestHandler.java:83)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:347)
        at org.elasticsearch.client.support.AbstractClient.fieldStats(AbstractClient.java:792)
        at org.elasticsearch.rest.action.fieldstats.RestFieldStatsAction.handleRequest(RestFieldStatsAction.java:70)
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:54)
        at org.elasticsearch.rest.RestController.executeHandler(RestController.java:207)
        at org.elasticsearch.rest.RestController$RestHandlerFilter.process(RestController.java:281)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:262)
        at org.elasticsearch.shield.rest.ShieldRestFilter.process(ShieldRestFilter.java:77)
        at org.elasticsearch.rest.RestController$ControllerFilterChain.continueProcessing(RestController.java:265)
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:176)
        at org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:128)
        at org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:86)
        at org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:348)
        at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:63)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        ...

(Victor Cano) #5

It seems like an issue with permissions because there isn't any problem when accessing with admin account.
The strange thing is my configuration is the same as the one I used for Shield 2.0.0 and as I read in the guide, there is no change in this part.

My roles.yml file contains the original data with the following part appended:

kibana4_type1_data:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    'logstash-type1-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    'logstash-type0-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

kibana4_type2_data:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    'logstash-type2-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    'logstash-type0-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

What I used to do with previous versions was configuring an index pattern such as logstash-* and user with roles kibana4_type1_data could access to logstash-type0-* and user with roles kibana4_type2_data could access to logstash-type1-* indices, while logstash-type0-* and logstash-type2-*. Now, it doesn't work. However, if I try to access to a, for example, index that follows the pattern logstash-type0-* with a user whose role is kibana4_type1_data, but it does work if I try to access to the specific index logstash-type0-2015.11.26.


(Victor Cano) #6

Regarding to the logs Elasticsearch writes, they are exactly the same as the ones JoachimKunz posted with the single exception in the index name.


(Jesus Gonzalez-Barahona) #7

I have the same symptoms, using Found. I have a role for anonymous access:

# A user with readonly access to dashboards.
anonymous:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:admin/get
    '.kibana':
      - indices:admin/exists
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

This was working until some days ago when I upgraded ElasticSearch to 2.1.0, which supposedly upgraded to Shield 2.1.0 as well.

And yes, it seems to have to do with permissions. Admin accounts are working for me too...


(Jay Modi) #8

The default role for Kibana users needs to have a permission added to allow the field stats api to be executed. Can you add indices:data/read/field_stats to the list of allowed privileges for the indices containing your data?


(Jesus Gonzalez-Barahona) #9

Thanks a lot! This worked like a charm. The configuration that is working for me is:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
    '.kibana':
      - indices:admin/exists
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

Maybe this could be mentioned/fixed in the Shield docs, in the chapter related to Kibana...


(Joachim Kunz) #10

Thanks a lot too! The configuration that is working for me is:

kibana4_team:
  cluster:
      - cluster:monitor/nodes/info
      - cluster:monitor/health
  indices:
    '*':
      privileges: indices:data/read/field_stats
    'team-*':
      privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get, indices:data/read/field_stats
    '.kibana':
      privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update

Without a global access to "indices: data/read/field_stats" it doesn't work!


(Jay Modi) #11

do you get the same error when only granting the privilege to the team-* pattern? And this is only on the discover page?


(CPS) #12

Hi all

It worked for me for one day. When I opened kibana today I got:

Discover: [index_not_found_exception] no such index, with: {"index":"[-*]"}

Have identical settings for anonymous just like you jgbarah and it doesn't work now !
that's really strange, my settings are:

elasticsearch.yml:

shield.authc.anonymous.authz_exception : false
shield.authc.anonymous.roles : anonymous
shield:
 authc:
   realms:
      esusers1:
        type: esusers
        order: 0

roles.yml:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
      - indices:data/write/bulk
    '.kibana':
      - indices:admin/exists
      - indices:data/write/bulk
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

Any ideas why anonymous access stopped working?


#13

I had the exact same problem. Installing Kibana 4.3.1 resolved this issue for me.


(CPS) #14

Continuing the discussion from Kibana doesn't work with wildcards since Shield 2.1.0:

Yes I agree, after installing 4.3.1 problem dissapeared. Thanks!
However, I had to add some privileges to the annonymous user. Now my roles settings for anonymous looks like following:

anonymous:
  cluster:
    - cluster:monitor/nodes/info
    - cluster:monitor/health
  indices:
    '*':
      - indices:admin/mappings/fields/get
      - indices:admin/validate/query
      - indices:data/read/search
      - indices:data/read/msearch
      - indices:data/read/field_stats
      - indices:admin/get
      - indices:data/write/bulk
      - indices:admin/create
      - indices:admin/template/get
    '.kibana':
      - indices:admin/template/get
      - indices:admin/exists
      - indices:data/write/bulk
      - indices:admin/mappings/fields/get
      - indices:admin/refresh
      - indices:admin/validate/query
      - indices:data/read/get
      - indices:data/read/mget
      - indices:data/read/search

ES 2.2: index_not_found_exception [logstash-*]
(Kennedy Kan) #15

I have come across with similar issue in the ES and Kibana 5.0 version, it seems like the problems on setting users and roles. But I have no ideas what is wrong. Can anyone help? Since it is another versio, thus I have opened a new post here.


(system) #16