Failed to load settings from [elasticsearch.yml]

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am getting the following exception . I have followed this link
https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-advanced-node-scheduling.html#k8s-define-elasticsearch-nodes-roles

root@k8smaster elastic]# ku logs quickstart-es-ingest-data-0 -n eckcluster
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: IllegalStateException[only value lists are allowed in serialized settings];
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1100)

Can anyone please help find the error , this is my yaml file

[root@k8smaster elastic]# vi elasticsearch.yaml
count: 3
config:
node.master: false
node.data: true
node.ingest: true
node.attr.data: hot
podTemplate:
spec:
containers:
- name: elasticsearch
resources:
limits:
memory: 16Gi
cpu: 4
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/instance-type
operator: In
values:
- highio
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: myStorage

2

This manifest doesn't seem to be complete. It is missing a few important fields like apiVersion and version and seems to be invalid because there are no master nodes configured at all. Please paste the full manifest and we can take a look. You can surround the text with ``` to format it properly.

4

'''
''' apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.8.1
nodeSets:

  • name: master
    count: 3
    config:
    node.master: true
    node.data: false
    node.ingest: false
    node.remote_cluster_client: false
  • name: ingest-data
    count: 3
    config:
    node.master: false
    node.data: true
    node.ingest: true
    node.attr.data: hot
    podTemplate:
    spec:
    containers:
    - name: elasticsearch
    resources:
    limits:
    memory: 16Gi
    cpu: 4
    affinity:
    nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
    - key: beta.kubernetes.io/instance-type
    operator: In
    values:
    - highio
    volumeClaimTemplates:
    • metadata:
      name: elasticsearch-data
      spec:
      accessModes:
      • ReadWriteOnce
        resources:
        requests:
        storage: 20Gi
        storageClassName: mystorage'''
5

Please check the above yaml file and let me know if there is a fix.

6

There are no obvious problems with the config. I was able to create a cluster successfully using your manifest. What are the outputs of the following commands?

kubectl get elasticsearch quickstart

kubectl get pods -l elasticsearch.k8s.elastic.co/cluster-name=quickstart

kubectl describe sts quickstart-es-ingest-data

kubectl logs quickstart-es-ingest-data-0
1 Like
7

Thank you for reply. However, the same error still persists.

I have used Elastic 7.9.0 this time with 2 data nodes.

[root@k8smaster elastic]# kubectl get pods -n ec
NAME READY STATUS RESTARTS AGE
quickstart-es-ingest-data-0 0/1 CrashLoopBackOff 7 12m
quickstart-es-ingest-data-1 0/1 CrashLoopBackOff 7 12m
quickstart-es-master-0 1/1 Running 0 12m

[root@k8smaster elastic]# kubectl get elasticsearch quickstart
Error from server (NotFound): elasticsearches.elasticsearch.k8s.elastic.co "quickstart" not found
[root@k8smaster elastic]# kubectl get pods -l elasticsearch.k8s.elastic.co/cluster-name=quickstart
No resources found in default namespace.
[root@k8smaster elastic]# kubectl describe sts quickstart-es-ingest-data
Error from server (NotFound): statefulsets.apps "quickstart-es-ingest-data" not found
[root@k8smaster elastic]# kubectl logs quickstart-es-ingest-data-0
Error from server (NotFound): pods "quickstart-es-ingest-data-0" not found
[root@k8smaster elastic]#

Error:
root@k8smaster elastic]# ku logs quickstart-es-ingest-data-0 -n ec
Exception in thread "main" SettingsException[Failed to load settings from [elasticsearch.yml]]; nested: IllegalStateException[only value lists are allowed in serialized settings];
at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1100)
at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1073)

8

You have to run the commands against the namespace the resources are in -- which seems to be ec in this case. What is the output of:

kubectl get secret quickstart-es-ingest-data-es-config -n ec -o=go-template='{{ index .data "elasticsearch.yml" | base64decode }}'
9

Output of the above command is

[root@k8smaster elastic]# kubectl get secret quickstart-es-ingest-data-es-config -n ec -o=go-template='{{ index .data "elasticsearch.yml" | base64decode }}'
cluster:
name: quickstart
discovery:
seed_providers: file
network:
host: 0.0.0.0
publish_host: {POD_IP} node: attr: data: hot data: true ingest: true master: false name: {POD_NAME}
path:
data: /usr/share/elasticsearch/data
logs: /usr/share/elasticsearch/logs
podTemplate:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/instance-type
operator: In
values:
- highio
containers:
- name: elasticsearch
resources:
limits:
cpu: 4
memory: 16Gi
xpack:
license:
upload:
types:
- trial
- enterprise
security:
authc:
realms:
file:
file1:
order: -100
native:
native1:
order: -99
reserved_realm:
enabled: "false"
enabled: "true"
http:
ssl:
certificate: /usr/share/elasticsearch/config/http-certs/tls.crt
certificate_authorities: /usr/share/elasticsearch/config/http-certs/ca.crt
enabled: true
key: /usr/share/elasticsearch/config/http-certs/tls.key
transport:
ssl:
certificate: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.crt
certificate_authorities:
- /usr/share/elasticsearch/config/transport-certs/ca.crt
- /usr/share/elasticsearch/config/transport-remote-certs/ca.crt
enabled: "true"
key: /usr/share/elasticsearch/config/node-transport-cert/transport.tls.key
verification_mode: certificate
[root@k8smaster elastic]#

10

It's very difficult to decipher this because it's not properly formatted (you can use triple backticks (```) as in Markdown to format it). The reason formatting is important is because it is YAML and spacing is significant.

I see a podTemplate definition amongst the contents of elasticsearch.yml. it shouldn't be there at all. My hunch is that in your Elasticsearch manifest, you have used the wrong spacing -- causing the podTemplate definition to become part of the config field. This explains the problem because Elasticsearch is seeing an invalid configuration file. Please correct the spacing in your manifest and the problem should go away.

11

You spotted correctly !! Thanks. I could get the yaml working, however the data pod is in pending state. I am debugging and will reach out to you if I need further help.