Having issues with more than 1 instance

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi there so I've gotten the single instance configuration to work. And Kibana is able to connect to it.

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elastic
spec:
  version: 7.4.2
  nodeSets:
    - name: default
      count: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
  http:
    tls:
      selfSignedCertificate:
        disabled: true

However as soon as I try to add more, in order to scale out reads e.g.:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elastic
spec:
  version: 7.4.2
  nodeSets:
    - name: default
      count: 3
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
  http:
    tls:
      selfSignedCertificate:
        disabled: true

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: elastic
spec:
  version: 7.4.2
  nodeSets:
    - name: default
      count: 1
      config:
        node.master: true
        node.data: true
        node.ingest: true
        node.store.allow_mmap: false
    - name: data
      count: 2
      config:
        node.master: false
        node.data: true
        node.ingest: false
        node.store.allow_mmap: false
  http:
    tls:
      selfSignedCertificate:
        disabled: true

All of my Elasticsearch pods start crashing/erroring out, and Kibana is not able to connect. If I start with 1 Elasticsearch + 1 Kibana it is able to connect, then when I scale up to 3 Elasticsearch, the pods stay running but eventually Kibana shows everything in a red state and it no longer works. I am running on an 8 vCPU x 32GB mem + 4 vCPU x 16GB mem + 4 vCPU x 16GB mem, so I believe I have enough resources.

Any suggestions?

2

Can you share the logs of the ES pods when they crash? That should help diagnose what is happening. I used your original spec and then increased the count to 3 and it worked as expected. I then tried applying the extra nodeset over it as in your third example and it also worked.

4

I was able to get another ES node up and it didn't crash, but I will provide logs if it does again.

However in my Kibana monitoring it still shows a yellow state, and only shows the original 1 node. It doesn't seem to be picking up the new node elastic-es-default-1.

elastic-es-default-0       Running
elastic-es-default-1       Running

I also noticed when I upgrade the count, it won't seem to go past 2. For example when I change to count: 3 and re-apply the yaml, it stays at 2 pods and nothing happens, no errors.

And in the Kibana I am intermittently getting "Internal Server Error". It seems to happen quite frequently but will eventually fix itself if you keep refreshing.

SearchError: Internal Server Error
    at https://localhost:5601/bundles/commons.bundle.js:5:1843343
    at processQueue (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:203810)
    at https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:204773
    at Scope.$digest (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:214757)
    at Scope.$apply (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:217761)
    at done (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:136089)
    at completeRequest (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:139882)
    at XMLHttpRequest.requestLoaded (https://localhost:5601/built_assets/dlls/vendors.bundle.dll.js:417:138624)

I am running on IBM Cloud Kubernetes. I don't see anything useful in any logs, but let me know if there's any specific info I can provide to help troubleshoot the issue.