Elastic didn't load properly. Check the server output for more information

Elastic Stack Kibana
1

Hi Team,

We have installed the Elasticsearch (8.5.1) and Kibana through helm following git hub link GitHub - elastic/helm-charts: You know, for Kubernetes. We are facing intermittent issue on Kibana Ui as " Elastic did not load properly. Check the server output for information". On checking the server output we see no logs related to this issue.

Which version of Elasticsearch and Kibana are you using? -->8.5.1

How is Elasticsearch configured? --->initial installation through helm

Are there any errors or warnings in the Elasticsearch logs? --> there are no errors in logs

How is Kibana configured? ---> Initial installation through helm

Are there any errors or warnings in the Kibana logs? --->No errors

What is the size and specification of the hardware hosting the components? --> The size and specification are bigger than values mentioned in GitHub deployment files (GitHub - elastic/helm-charts: You know, for Kubernetes).

What type of hardware is used? Kubernetes as service cluster
Kubernetes version --> client version v1.21.8, server version -->1.26.6

Note: * The issue was reproduceable with ECK v8.5.1 and v7.14.0.

  • Our use case is to make use of default basic security available in the stable version v8.5.1.
    We also need information on how to enable debug logs on elastic and Kibana deployment.

Thanks & Regards,
Sarala

2

Hi Sarala,

Could you post the ECK manifest used to deploy the cluster?

Or did you use the helm chart? Please note that the helm charts are deprecated, and you must use ECK to deploy the cluster.

Regards,
Nachiket

3

GitHub - elastic/helm-charts: You know, for Kubernetes => Deprecated

This is the alternative => Deploy ECK in your Kubernetes cluster | Elastic Cloud on Kubernetes [2.9] | Elastic

4

Hi @NerdSec ,

We have seen same issue eck(2.9) as well,
Please find the elastic and kibana manifest files below.
Elastic manifest file

<apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 8.5.1
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
podTemplate:
spec:
containers:
- name: elasticsearch
env:
- name: ES_JAVA_OPTS
value: "-Xms2g -Xmx2g"
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: 2
memory: 4Gi
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
storageClassName: network-block
volumeMode: Filesystem
http:
tls:
certificate:
secretName: elastic-certificates>

kibana manifest file:

</ apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
spec:
version: 8.5.1
count: 1
elasticsearchRef:
name: elasticsearch
http:
tls:
certificate:
secretName: elastic-certificates />

Thanks&Regards,
Sarala K

5

Have you increased the max map count?

Could also post the logs for Elasticsearch pod? How are you deploying the worker nodes? Which K8S service are you using?

7

Hi @NerdSec ,

No errors in the pod logs .

Thanks&Regards,
Sarala k

8

Hi Team,
Please find the pod logs for elk.
@timestamp":"2023-08-04T23:35:26.565Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-Country.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#5]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-08-06T22:04:21.601Z", "log.level": "WARN", "message":"health check of [/usr/share/elasticsearch/data] took [5603ms] which is above the warn threshold of [5s]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#1]","log.logger":"org.elasticsearch.monitor.fs.FsHealthService","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-08-07T23:35:28.907Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-12002610813008231402/geoip-databases//GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#9]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-08-07T23:35:28.909Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-ASN.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#9]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-08-07T23:35:36.548Z", "log.level": "INFO", "message":"evicted [0] entries from cache after reloading database [/tmp/elasticsearch-12002610813008231402/geoip-databases/AO1d_sxLTcuGoVRFddrSOA/GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#5]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseReaderLazyLoader","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}
{"@timestamp":"2023-08-07T23:35:36.548Z", "log.level": "INFO", "message":"successfully loaded geoip database file [GeoLite2-City.mmdb]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-es-default-0][generic][T#5]","log.logger":"org.elasticsearch.ingest.geoip.DatabaseNodeService","elasticsearch.cluster.uuid":"","elasticsearch.node.id":"","elasticsearch.node.name":"elasticsearch-es-default-0","elasticsearch.cluster.name":"elasticsearch"}

Thanks&Regards,
Sarala k

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.