Bootstrap check failed in es 7.10.1

f4ct0r · January 14, 2021, 6:38am

Hey guys, I'm using kubernetes to deploy Elasticsearch 7.10.1.
at first, my elasticsearch cluster running perfect.
but when i restart the pods of elasticsearch, the elasticsearch cluster going error:

ERROR: [1] bootstrap checks failed
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-cluster.log

i try to figure it out the reason why the cluster starting failure, but the log file /usr/share/elasticsearch/logs/elasticsearch-cluster.log dose not exist.

can anyone point me?

f4ct0r · January 14, 2021, 6:39am

my kubernetes configure like this:

---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  labels:
    elastic-app: elasticsearch
  name: elasticsearch-cluster
spec:
  replicas: 3
  revisionHistoryLimit: 10
  serviceName: elasticsearch-service
  selector:
    matchLabels:
      elastic-app: elasticsearch
  template:
    metadata:
      labels:
        elastic-app: elasticsearch
    spec:
      containers:
        - name: elasticsearch-cluster
          image: elasticsearch:7.10.1
          imagePullPolicy: IfNotPresent
          lifecycle:
            postStart:
              exec:
                command: [ "/bin/bash", "-c", "sysctl -w vm.max_map_count=262144; ulimit -l unlimited;" ]
          ports:
            - containerPort: 9200
              protocol: TCP
            - containerPort: 9300
              protocol: TCP
          env:
            - name: "cluster.name"
              value: "elasticsearch-cluster"
            - name: "discovery.seed_hosts"
              value: "elasticsearch-service"
            - name: "cluster.initial_master_nodes"
              value: "elasticsearch-cluster-0,elasticsearch-cluster-1,elasticsearch-cluster-2"
            - name: "node.max_local_storage_nodes"
              value: "9"
            - name: "node.master"
              value: "true"
            - name: "node.data"
              value: "true"
            - name: "node.ingest"
              value: "true"
            - name: "http.cors.enabled"
              value: "true"
            - name: "http.cors.allow-origin"
              value: "*"
            - name: "xpack.license.self_generated.type"
              value: "basic"
            - name: "xpack.security.enabled"
              value: "true"
            - name: "xpack.security.transport.ssl.enabled"
              value: "false"
            - name: "xpack.monitoring.enabled"
              value: "true"
            - name: "ES_JAVA_OPTS"
              value: "-Xms32g -Xmx32g"
          securityContext:
            privileged: true
          volumeMounts:
            - mountPath: /usr/share/elasticsearch/data
              name: esdata
      volumes:
        - name: esdata
          hostPath:
            path: /siem-data/elasticsearch
            type: Directory
      tolerations:
        - key: node-role.kubernetes.io/elasticsearch-cluster
          effect: NoSchedule
      nodeSelector:
        app: es

---
kind: Service
apiVersion: v1
metadata:
  labels:
    elastic-app: elasticsearch
  name: elasticsearch-service
spec:
  ports:
    - port: 9200
      name: eshttp
      targetPort: 9200
      nodePort: 39200
    - port: 9300
      name: transport
      targetPort: 9300
      nodePort: 39300
  type: NodePort
  selector:
    elastic-app: elasticsearch

f4ct0r · January 14, 2021, 6:40am

and the log from kubernetes stdout like this:

{"type": "deprecation", "timestamp": "2021-01-14T06:13:15,095Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[node.data] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "deprecation", "timestamp": "2021-01-14T06:13:15,121Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[node.max_local_storage_nodes] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "server", "timestamp": "2021-01-14T06:13:15,145Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda4)]], net usable_space [4.8tb], net total_space [4.8tb], types [xfs]" }
{"type": "server", "timestamp": "2021-01-14T06:13:15,146Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "heap size [32gb], compressed ordinary object pointers [false]" }
{"type": "deprecation", "timestamp": "2021-01-14T06:13:15,155Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "deprecation", "timestamp": "2021-01-14T06:13:15,167Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[node.ingest] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "server", "timestamp": "2021-01-14T06:13:15,326Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "node name [elasticsearch-cluster-0], node ID [rIVpY1eBT_iDYgL4SlVa2g], cluster name [elasticsearch-cluster], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]" }
{"type": "deprecation", "timestamp": "2021-01-14T06:13:19,837Z", "level": "DEPRECATION", "component": "o.e.d.c.s.Settings", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[xpack.monitoring.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version." }
{"type": "server", "timestamp": "2021-01-14T06:13:22,338Z", "level": "INFO", "component": "o.e.x.m.p.l.CppLogMessageHandler", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "[controller/175] [Main.cc@114] controller (64 bit): Version 7.10.1 (Build 11e1ac84105757) Copyright (c) 2020 Elasticsearch BV" }
{"type": "server", "timestamp": "2021-01-14T06:13:23,338Z", "level": "INFO", "component": "o.e.x.s.a.s.FileRolesStore", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "parsed [0] roles from file [/usr/share/elasticsearch/config/roles.yml]" }
{"type": "server", "timestamp": "2021-01-14T06:13:25,159Z", "level": "INFO", "component": "o.e.t.NettyAllocator", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]" }
{"type": "server", "timestamp": "2021-01-14T06:13:25,298Z", "level": "INFO", "component": "o.e.d.DiscoveryModule", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "using discovery type [zen] and seed hosts providers [settings]" }
{"type": "server", "timestamp": "2021-01-14T06:13:26,265Z", "level": "WARN", "component": "o.e.g.DanglingIndicesState", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually" }
{"type": "server", "timestamp": "2021-01-14T06:13:27,069Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "initialized" }
{"type": "server", "timestamp": "2021-01-14T06:13:27,070Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "starting ..." }
{"type": "server", "timestamp": "2021-01-14T06:13:27,268Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "publish_address {192.168.4.124:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2021-01-14T06:13:27,715Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
ERROR: [1] bootstrap checks failed
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/elasticsearch-cluster.log
{"type": "server", "timestamp": "2021-01-14T06:13:27,778Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "stopping ..." }
{"type": "server", "timestamp": "2021-01-14T06:13:27,801Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "stopped" }
{"type": "server", "timestamp": "2021-01-14T06:13:27,801Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "closing ..." }
{"type": "server", "timestamp": "2021-01-14T06:13:27,825Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "closed" }
{"type": "server", "timestamp": "2021-01-14T06:13:27,829Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "elasticsearch-cluster", "node.name": "elasticsearch-cluster-0", "message": "Native controller process has stopped - no new native processes can be started" }

DavidTurner · January 14, 2021, 7:03am

A description of the problem and two suggestions for how to fix it are both here:

f4ct0r · January 14, 2021, 9:38am

Yeap, I’ve already noticed that, but thank you anyway

system · February 11, 2021, 9:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.