I've been having some difficulty following the ECK Quickstart.
To begin with I install the operator and instantiate a single node instance:
[root@a0002-flexnet ~]# kubectl apply -f https://download.elastic.co/downloads/eck/0.8.1/all-in-one.yaml
[SNIP]
[root@a0002-flexnet ~]# cat <<EOF | kubectl apply -f -
> apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
> kind: Elasticsearch
> metadata:
> name: quickstart
> spec:
> version: 7.1.0
> nodes:
> - nodeCount: 1
> config:
> node.master: true
> node.data: true
> node.ingest: true
> EOF
elasticsearch.elasticsearch.k8s.elastic.co/quickstart created
[root@a0002-flexnet ~]# kubectl get elasticsearches.elasticsearch.k8s.elastic.co
NAME HEALTH NODES VERSION PHASE AGE
quickstart red 7.1.0 Pending 35s
[root@a0002-flexnet ~]# kubectl get elasticsearches.elasticsearch.k8s.elastic.co
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 1 7.1.0 Operational 81s
At this point I can open a connection to the quickstart-es service and authenticate correctly.
So far so good!
A describe reveals that it looks healthy:
[root@a0002-flexnet ~]# kubectl describe elasticsearches.elasticsearch.k8s.elastic.co quickstart
Name: quickstart
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"elasticsearch.k8s.elastic.co/v1alpha1","kind":"Elasticsearch","metadata":{"annotations":{},"name":"quickstart","namespace":...
API Version: elasticsearch.k8s.elastic.co/v1alpha1
Kind: Elasticsearch
Metadata:
Creation Timestamp: 2019-07-17T20:21:03Z
Finalizers:
expectations.finalizers.elasticsearch.k8s.elastic.co
observer.finalizers.elasticsearch.k8s.elastic.co
secure-settings.finalizers.elasticsearch.k8s.elastic.co
licenses.finalizers.elasticsearch.k8s.elastic.co
Generation: 2
Resource Version: 37578333
Self Link: /apis/elasticsearch.k8s.elastic.co/v1alpha1/namespaces/default/elasticsearches/quickstart
UID: 661930de-a8d0-11e9-9f84-ac1f6b7678a2
Spec:
Http:
Service:
Metadata:
Spec:
Tls:
Nodes:
Config:
Node . Data: true
Node . Ingest: true
Node . Master: true
Node Count: 1
Pod Template:
Metadata:
Creation Timestamp: <nil>
Spec:
Containers: <nil>
Update Strategy:
Version: 7.1.0
Status:
Available Nodes: 1
Cluster UUID: GguK3wIwSAe_W2hWWIiVsg
Health: green
Master Node: quickstart-es-gbkkpdr7lm
Phase: Operational
Service: quickstart-es
Zen Discovery:
Minimum Master Nodes: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 11m elasticsearch-controller Created pod quickstart-es-gbkkpdr7lm
Normal StateChange 10m elasticsearch-controller Master node is now quickstart-es-gbkkpdr7lm
Here's where things go off the rails a bit:
After about 10 minutes, the elasticsearch pod falls over.
{"type": "server", "timestamp": "2019-07-17T20:32:22,527+0000", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "quickstart", "node.name": "quickstart-es-gbkkpdr7lm", "cluster.uuid": "GguK3wIwSAe_W2hWWIiVsg", "node.id": "pUETgQReRNuWE0mvNi6q-A", "message": "Native controller process has stopped - no new native processes can be started" }
{"level":"info","ts":1563395552.5241575,"logger":"process-manager","msg":"Update process state","action":"terminate","id":"es","state":"failed","pid":15}
{"level":"info","ts":1563395552.531126,"logger":"process-manager","msg":"HTTP server closed"}
{"level":"info","ts":1563395552.5324,"logger":"process-manager","msg":"Exit","reason":"process failed","code":-1}
Afterwards, the operator shows the service is degraded and it never recovers:
[root@a0002-flexnet ~]# kubectl describe elasticsearches.elasticsearch.k8s.elastic.co quickstart
Name: quickstart
Namespace: default
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"elasticsearch.k8s.elastic.co/v1alpha1","kind":"Elasticsearch","metadata":{"annotations":{},"name":"quickstart","namespace":...
API Version: elasticsearch.k8s.elastic.co/v1alpha1
Kind: Elasticsearch
Metadata:
Creation Timestamp: 2019-07-17T20:21:03Z
Finalizers:
expectations.finalizers.elasticsearch.k8s.elastic.co
observer.finalizers.elasticsearch.k8s.elastic.co
secure-settings.finalizers.elasticsearch.k8s.elastic.co
licenses.finalizers.elasticsearch.k8s.elastic.co
Generation: 2
Resource Version: 37583256
Self Link: /apis/elasticsearch.k8s.elastic.co/v1alpha1/namespaces/default/elasticsearches/quickstart
UID: 661930de-a8d0-11e9-9f84-ac1f6b7678a2
Spec:
Http:
Service:
Metadata:
Spec:
Tls:
Nodes:
Config:
Node . Data: true
Node . Ingest: true
Node . Master: true
Node Count: 1
Pod Template:
Metadata:
Creation Timestamp: <nil>
Spec:
Containers: <nil>
Update Strategy:
Version: 7.1.0
Status:
Cluster UUID: GguK3wIwSAe_W2hWWIiVsg
Health: red
Master Node: quickstart-es-gbkkpdr7lm
Phase: Pending
Service: quickstart-es
Zen Discovery:
Minimum Master Nodes: 1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 11m elasticsearch-controller Created pod quickstart-es-gbkkpdr7lm
Normal StateChange 10m elasticsearch-controller Master node is now quickstart-es-gbkkpdr7lm
Warning Unhealthy 5s elasticsearch-controller Elasticsearch cluster health degraded
After some time the pod goes into Waiting: CrashLoopBackOff.
How do I start to troubleshoot this?
What would cause this single-instance test cluster to crash reliably after 10 minutes?
Many thanks!
-Z