[undefined] Forbidden error when accessing Index Management in Kibana 6.7.1

Hi,
In our production infrastructure we are trying to access pages under the Management section in Kibana 6.7.1. Specifically, we are receiving the error [undefined] Forbidden when trying to access the Index Management webpage (and the same errore as in the Index Lifecycle management as well).

The elasticsearch.yml configuration is here included. From the documentation, it is unclear to me whether the kibana_access: rw permission level actually allows to access Index Management webpage. Before toggling settings in a production environment, a support on this would be very helpful.

    readonlyrest:
      enable: true # optional, defaults=true if at least 1 "access_control_rules" block
      audit_collector: true

      ssl:
        enable: true
        keystore_file: "/etc/elasticsearch/ssl/server.pfx"
        keystore_pass: fe4f5g6h7j8kj7hg
        key_pass: d34f5g6h7jh6g5f4d

      response_if_req_forbidden: Forbidden

      access_control_rules:

      - name: "::LOGSTASH::"
        # auth_key is good for testing, but replace it with `auth_key_sha1`!
        auth_key_sha256: 9d4324c242434cc2323c444243c234c248
        actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
        indices: ["logstash*","security*","stores*","ourappname1*","jmeter*",'auditbeat*']
        verbosity: error # don't log successful request

      # We trust Kibana's server side process, full access granted via HTTP authentication
      - name: "::KIBANA-SRV::"
        # auth_key is good for testing, but replace it with `auth_key_sha256`!
        auth_key_sha256: a71c3eff652c424c234c234c24c224c234c0d439543df
        verbosity: error # don't log successful request

      - name: "::CURATOR::"
        # auth_key is good for testing, but replace it with `auth_key_sha256`!
        auth_key_sha256: 3bf6d8438d2cec56b4647b47b465v25543a15ed8aa52440
        verbosity: error # don't log successful request

      - name: "::GRAFANA-SRV::"
        # auth_key is good for testing, but replace it with `auth_key_sha256`!
        auth_key_sha256: 16ed43v54v455v36767b474b92c477d5a65541
        actions: ["indices:data/read/*","indices:admin/get","cluster:monitor/main","indices:admin/mappings/get"]
        verbosity: error # don't log successful request

      - name: "::monitoring::"
        auth_key_sha256: bbd56113efc207504v53v5345v36v36536b3749fb47414a8971695
        verbosity: error
        actions: ["cluster:monitor/*","indices:monitor/*"]

      - name: "::elastalert_on_own_index::"
        auth_key_sha256: 69db943c4f90ca37b48n75v36468n7b7v3cr3f19c198b99367661
        actions: ["cluster:monitor/main","indices:admin/types/exists","indices:data/read/*","indices:data/write/*","indices:admin/template/*","indices:admin/create"]
        verbosity: error
        indices: ["elastalert*"]

      - name: "::elastalert_on_logstash::"
        auth_key_sha256: 69db943c4v367b66n578m6nb73v63534c5c198b99367661
        verbosity: error
        actions: ["indices:data/read/*","cluster:monitor/*"]
        indices: ["logstash*","security*","auditbeat*"]

      - name: "::es2logs_on_logstash::"
        auth_key_sha256: 290366bac543v56b7n8m9m7nb6v53c44b5cb7eb33
        verbosity: error
        actions: ["indices:data/read/*","cluster:monitor/*"]
        indices: ["logstash*"]

      - name: "field_caps stuff"
        verbosity: error # don't log successful request
        type: allow
        actions: ["indices:data/read/field_caps"]
        hosts: ["ouripsubnet1","ourprivatesubnet2"]

      - name: "internal access to kibana index"
        verbosity: error # don't log successful request
        type: allow
        hosts: [“ouripsubnet1”,”ouripsubnet2”]
        actions: ["indices:data/read/search","indices:data/read/get","indices:data/read/mget"]
        indices: [".kibana"]

      - name: "kibana_rw"
        kibana_access: rw
        indices: [".kibana", "stores*", "logstash*”,”ourdomainname*”,”anotherdomainname*”]
        ldap_authentication:
          name: "ldap1"
          cache_ttl_in_sec: 60
        ldap_authorization:
          name: "ldap1"
          groups: ["kibana_rw"]
          cache_ttl_in_sec: 60

      - name: "elasticsearch_adm"
        ldap_authentication:
          name: "ldap1"
          cache_ttl_in_sec: 60
        ldap_authorization:
          name: "ldap1"
          groups: ["elasticsearch_adm"]
          cache_ttl_in_sec: 60

      - name: "kibana_ro"
        kibana_access: ro
        indices: [".kibana", "stores*", "logstash*”,”domainname1*”,”domainname2*”]
        ldap_authentication:
          name: "ldap1"
          cache_ttl_in_sec: 60
        ldap_authorization:
          name: "ldap1"
          groups: ["kibana_ro"]
          cache_ttl_in_sec: 60
      ldaps:

      - name: ldap1
        host: "ldap.prod.ourdomainname”
        port:  636                                                # default 389
        ssl_enabled: true                                    # default true
        ssl_trust_all_certs: true                                 # default false
        search_user_base_DN: "ou=users,dc=prod,dc=domainname”
        user_id_attribute: "uid"                                  # default "uid"
        search_groups_base_DN: "ou=groups,dc=prod,dc=domainname”
        unique_member_attribute: "member"                   # default "uniqueMember"
        connection_pool_size: 10                                  # default 30
        connection_timeout_in_sec: 10                             # default 1
        request_timeout_in_sec: 10                                # default 1
        cache_ttl_in_sec: 60

You are gonna need manage-ilm and `manage_index_templates" at the minimum to use it, as far as I know, as it is a very specific part of Kibana that can delete and modify indices.

Hi Marius, thank you for supporting.

It is not clear to me where I should insert manage-ilm and manage_index_templates, would you elaborate on that ?

In the meanwhile, I added a specific role called "kibana-admin" on the ldap side and, based on this, I added the following to the readonlyrest configuration:

      - name: "kibana_admin"
        kibana_access: admin
        indices: ["admin/ilm/explain", "monitor/settings/get", "monitor/stats"]
        ldap_authentication:
          name: "ldap1"
          cache_ttl_in_sec: 60
        ldap_authorization:
         name: "ldap1"
         groups: ["kibana_admin"]
         cache_ttl_in_sec: 60

After restarting both elasticsearch and kibana, we are still unable to access the Index Management page. From elasticsearch.log I see nothing suspicious except a warning which seems uncorrelated to me:

    [2020-03-26T11:41:50,620][WARN ][r.suppressed             ] [OJLMpVY] path: /.kibana/_search, params: {rest_total_hits_as_int=true, size=1000, index=.kibana, from=0}
    org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
    	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:291) ~[elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:100) ~[elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:208) ~[elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:187) [elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.1.jar:6.7.1]
    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.1.jar:6.7.1]
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
    	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
    [2020-03-26T11:42:00,514][INFO ][o.e.m.j.JvmGcMonitorService] [OJLMpVY] [gc][75] overhead, spent [362ms] collecting in the last [1s]
    [2020-03-26T11:42:09,117][INFO ][o.e.c.r.a.AllocationService] [OJLMpVY] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[releases][4], [releases][3]] ...]).
    [2020-03-26T11:55:33,937][INFO ][o.e.c.m.MetaDataIndexTemplateService] [OJLMpVY] adding template [.management-beats] for index patterns [.management-beats]
    [2020-03-26T11:55:40,245][INFO ][o.e.c.m.MetaDataIndexTemplateService] [OJLMpVY] adding template [kibana_index_template:.kibana] for index patterns [.kibana]
    [2020-03-26T12:10:04,121][INFO ][o.e.c.s.ClusterSettings  ] [OJLMpVY] updating [cluster.routing.allocation.enable] from [primaries] to [all]

In manually enabled kibana.log I see no warnings nor errors.