Elasticsearch and Kibana 8.3.3 basic setup not working in Azure Kubernetes

Hi,
I have installed ECK with the CRDS 2.3.0 (from the main documentation) and proceeded to install elasticsearch and kibana via the manifests provided in the documentation. After deploying the Kibana logs, I am seeing multiple errors reported.
The main error seems sto be with the plugins.security.authorization.

Please check below the manifests:

# Elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  version: 8.3.3
  nodeSets:
  - name: master-nodes
    count: 1
    config:
      node.roles: ["master"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi
        storageClassName: elasticsearch-azurefile
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        nodeSelector:
          agentpool: agentpool2
  - name: data-nodes
    count: 3
    config:
      node.roles: ["data", "ingest", "transform"]
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1000Gi
        storageClassName: elasticsearch-azurefile
    podTemplate:
      spec:
        initContainers:
        - name: sysctl
          securityContext:
            privileged: true
            runAsUser: 0
          command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
        nodeSelector:
          agentpool: agentpool2
---
# Kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  version: 8.3.3
  count: 1
  elasticsearchRef:
    name: elasticsearch
  podTemplate:
    spec:
      containers:
      - name: kibana
        env:
          - name: NODE_OPTIONS
            value: "--max-old-space-size=2048"
        resources:
          requests:
            memory: 1Gi
            cpu: 0.5
          limits:
            memory: 2.5Gi
            cpu: 2
      nodeSelector:
        agentpool: agentpool2

Error in the logs:

2022-08-12T18:39:16.891+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [index_not_found_exception] Reason: no such index [.security]
[2022-08-12T18:39:16.985+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 2 in 4 seconds.
[2022-08-12T18:39:16.985+00:00][INFO ][savedobjects-service] [.kibana_task_manager] INIT -> INIT. took: 2006ms.
[2022-08-12T18:39:20.991+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with 'security_exception: [index_not_found_exception] Reason: no such index [.security]'. Retrying attempt 3 in 8 seconds
.....
....
[2022-08-12T18:39:59.816+00:00][INFO ][plugins.securitySolution.endpoint:metadata-check-transforms-task:0.0.1] no endpoint installation found
[2022-08-12T18:39:59.852+00:00][INFO ][status] Kibana is now available (was degraded)
[2022-08-12T18:40:02.253+00:00][INFO ][plugins.reporting.store] Creating ILM policy for managing reporting indices: kibana-reporting
[2022-08-12T18:40:11.551+00:00][INFO ][plugins.ruleRegistry] Installed resources for index .preview.alerts-security.alerts
[2022-08-12T18:40:13.131+00:00][INFO ][status] Kibana is now degraded (was available)
[2022-08-12T18:40:18.116+00:00][INFO ][status] Kibana is now available (was degraded)
[2022-08-12T18:40:23.337+00:00][INFO ][plugins.ml] Task ML:saved-objects-sync-task: 1 ML saved object synced
[2022-08-12T18:40:23.349+00:00][INFO ][plugins.fleet] Fleet setup completed
[2022-08-12T18:40:23.362+00:00][INFO ][plugins.securitySolution] Dependent plugin setup complete - Starting ManifestTask
[2022-08-12T18:41:40.766+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: Request timed out
[2022-08-12T18:43:41.056+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: Request timed out
[2022-08-12T18:44:39.453+00:00][WARN ][plugins.usageCollection.usage-collection.collector-set] ResponseError: security_exception: [security_exception] Reason: action [indices:data/read/search] is unauthorized for service account [elastic/kibana], this action is granted by the index privileges [read,all]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:476:27)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)
    at ClientTraced.SearchApi [as search] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/search.js:65:12)
    at fetchClusters (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/alerts/fetch_clusters.js:53:20)
    at UsageCollector.fetch (/usr/share/kibana/x-pack/plugins/monitoring/server/kibana_monitoring/collectors/get_usage_collector.js:103:24)
    at CollectorSet.fetchCollector (/usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:146:24)
    at fetch_monitoring (/usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:175:107)
    at /usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:176:16
    at async Promise.all (index 46)
    at Object.bulkFetch (/usr/share/kibana/src/plugins/usage_collection/server/collector/collector_set.js:174:31)
    at getKibana (/usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_kibana.js:65:17)
    at async Promise.all (index 3)
    at /usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_local_stats.js:78:76
    at async Promise.all (index 0)
    at getLocalStats (/usr/share/kibana/src/plugins/telemetry/server/telemetry_collection/get_local_stats.js:77:10)
    at async Promise.all (index 0)
    at Object.getStatsWithXpack [as statsGetter] (/usr/share/kibana/x-pack/plugins/telemetry_collection_xpack/server/telemetry_collection/get_stats_with_xpack.js:27:48)
    at TelemetryCollectionManagerPlugin.getUsageForCollection (/usr/share/kibana/src/plugins/telemetry_collection_manager/server/plugin.js:279:19)
    at TelemetryCollectionManagerPlugin.getStats (/usr/share/kibana/src/plugins/telemetry_collection_manager/server/plugin.js:213:29)
    at FetcherTask.fetchTelemetry (/usr/share/kibana/src/plugins/telemetry/server/fetcher.js:192:12)
    at FetcherTask.sendIfDue (/usr/share/kibana/src/plugins/telemetry/server/fetcher.js:88:18)
[2022-08-12T18:44:39.458+00:00][WARN ][plugins.usageCollection.usage-collection.collector-set] Unable to fetch data from monitoring collector

Please any help is much appreciated, I've been stuck for days.

I have moved your question to the Kibana forum in the hope that someone with more Kibana expertise might be able to help you with your problem.

Actually, reading through the error message again I am wondering if your Elasticsearch cluster is healthy? Have you checked that? You can do that by either using kubectl

kubectl get es elasticsearch

Or by calling the Elasticsearch API directly

GET _cat/health

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.