ECK Kibana recieves 401 in minutes if no operations in Kibana

Elastic Stack Kibana
1

Hi, I used the ECK operator to deploy an ELK cluster on my local k8s cluster with:

  • 3 instances of Elasticsearch
  • 3 instances of Logstash
  • 1 Kibana instance
  • A few Filebeat instances

The ELK pipeline is working well—Filebeat ships logs to Logstash, which then sends them to Elasticsearch. I can view the data and run queries in Kibana. However, after logging into Kibana and staying idle for a few minutes, I get the error:
"An unexpected authentication error occurred. Please log in again."

What could be causing this issue?

image
image788×588 18.8 KB

These are the manifests that I used to create the cluster.

Elasticsearch:

apiVersion: v1
kind: Secret
metadata:
  name: eck-test-env-es-elastic-user
  namespace: elk
type: Opaque
data:
  elastic: cm9nZXJmYW4=
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: eck-test-env
  namespace: elk
spec:
  version: 8.17.3
  image: 192.168.38.89:30003/elk/elasticsearch/elasticsearch:8.17.3
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
      - metadata:
          name: elasticsearch-data
        spec:
          accessModes:
            - ReadWriteOnce
          storageClassName: eck-local-storage
          resources:
            requests:
              storage: 300Gi
  transport:
    service:
      spec:
        type: NodePort
        ports:
          - name: transport
            port: 9200
            targetPort: 9200
            nodePort: 30011

Kibana:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: eck-test-env
  namespace: elk
spec:
  version: 8.17.3
  image: 192.168.38.89:30003/elk/kibana/kibana:8.17.3
  count: 1
  elasticsearchRef:
    name: eck-test-env
---
apiVersion: v1
kind: Service
metadata:
  name: eck-test-env-kb-nodeport
  namespace: elk
spec:
  type: NodePort
  ports:
    - port: 5601
      targetPort: 5601
      nodePort: 30012
      protocol: TCP
  selector:
      common.k8s.elastic.co/type: kibana
      kibana.k8s.elastic.co/name: eck-test-env
2

Here are some logs

Elasticsearch
Elasticsearch instances do not generate logs at this time period.

Kibana

[2025-03-26T01:37:34.737+00:00][INFO ][plugins.security.authentication] Performing login attempt with "basic" provider.
[2025-03-26T01:37:35.991+00:00][INFO ][plugins.security.authentication] Login attempt with "basic" provider succeeded (requires redirect: true).
[2025-03-26T01:40:03.088+00:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:1.0.0] [runTask()] started
[2025-03-26T01:40:03.088+00:00][WARN ][plugins.actions] Missing required project id while running actions:connector_usage_reporting
[2025-03-26T01:40:03.143+00:00][INFO ][plugins.fleet.fleet:delete-unenrolled-agents-task:1.0.0] [DeleteUnenrolledAgentsTask] runTask ended: Delete unenrolled agents is disabled
[2025-03-26T01:40:06.566+00:00][INFO ][plugins.fleet] Running Fleet Usage telemetry send task
[2025-03-26T01:40:13.585+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started
[2025-03-26T01:40:13.639+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success
[2025-03-26T01:40:14.734+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"eck-test-env"}
[2025-03-26T01:41:00.117+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:00.196+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2025-03-26T01:41:00.196+00:00][INFO ][plugins.fleet] Cleaning old indices
[2025-03-26T01:41:02.079+00:00][INFO ][plugins.fleet] Fleet setup completed
[2025-03-26T01:41:22.420+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:30.224+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:30.282+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2025-03-26T01:41:30.282+00:00][INFO ][plugins.fleet] Cleaning old indices
[2025-03-26T01:41:31.402+00:00][INFO ][plugins.fleet] Fleet setup completed
[2025-03-26T01:41:34.710+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:38.865+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:46.012+00:00][INFO ][plugins.fleet] Secrets storage is disabled as minimum fleet server version has not been met
[2025-03-26T01:41:47.881+00:00][INFO ][savedobjects-service.importer] Successfully imported 4 saved objects.



[2025-03-26T01:50:13.640+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started
[2025-03-26T01:50:13.703+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success
[2025-03-26T01:55:14.814+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"eck-test-env"}
[2025-03-26T02:00:13.754+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started
[2025-03-26T02:00:13.818+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success
[2025-03-26T02:10:13.833+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [runTask()] started
[2025-03-26T02:10:13.894+00:00][INFO ][plugins.fleet.fleet:unenroll-inactive-agents-task:1.0.0] [UnenrollInactiveAgentsTask] runTask ended: success
[2025-03-26T02:10:14.905+00:00][INFO ][plugins.fleet] Fleet Usage: {"agents_enabled":true,"agents":{"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"inactive":0,"unenrolled":0,"total_all_statuses":0,"updating":0},"fleet_server":{"total_all_statuses":0,"total_enrolled":0,"healthy":0,"unhealthy":0,"offline":0,"updating":0,"inactive":0,"unenrolled":0,"num_host_urls":0},"license_issued_to":"eck-test-env"}