Elasticsearch cluster fails to start

NK2812 · December 21, 2020, 2:19pm

Hi,

We use the eck operators to create the Elasticsearch and we create the cluster dynamically when there is a request to create the same.

we are using 7.5.0 version and trying to upgrade to 7.9.2 version. So when we try to create the cluster it is failing with below error logs

Starts Elasticsearch

Option Description

-E Configure a setting
-V, --version Prints Elasticsearch version information and exits
-d, --daemonize Starts Elasticsearch in the background
-h, --help Show help
-p, --pidfile Creates a pid file in the specified path on start
-q, --quiet Turns off standard output/error streams logging in console
-s, --silent Show minimal output
-v, --verbose Show verbose output
ERROR: setting [xpack.security.enabled] already set, saw [false] and [false]

we pass the flag xpack.security.enabled as false using the environment variable. we want this to be false only, by default it is true so we want to set it as flase.

Please guide how to fix this issue.

TimV · December 21, 2020, 11:03pm

ECK does not allow you to turn off security, if you really want a cluster that does not require authentication (which is not a good idea) then you can enable anonymous access.

NK2812 · December 22, 2020, 6:48am

@TimV Thanks for the reply.

We are using the readonly Rest plugin on top of the Elasticsearch, we install this plugin in the docker image and operators use that docker image to create the cluster.

We are setting xpack.security.enabled as false, if I don't set this flag then cluster fails with the error message saying Can not have more than 1 REST wrapper implementation.

It is working fine currently and we are using elasticsearch 7.5.0 version but failing when we are trying to upgrade it to 7.9.2 version.

NK2812 · December 22, 2020, 6:14pm

Hi @TimV

It is failing because of the docker-entrypoint.sh script, we need to run few commands when container starts, so we have added those in docker-entrypoint.sh script. so I tried removing those commands but still it didn't work with ES 7.9.2 version but works with 7.5.0 version.

below is the docker-entrypoint.sh file

#!/bin/bash

set -e

#Files created by Elasticsearch should always be group writable too
umask 0002

run_as_other_user_if_needed() {
if [[ "(id -u)" == "0" ]]; then # If running as root, drop to specified UID and run command exec chroot --userspec=1000 / "{@}"
else
# Either we are running in Openshift with random uid and are a member of the root group
# or with a custom --user
exec "${@}"
fi
}

#Allow user specify custom CMD, maybe bin/elasticsearch itself
#for example to directly specify -E style parameters for elasticsearch on k8s
#or simply to run /bin/bash to check the image
if [[ "1" != "eswrapper" ]]; then if [[ "(id -u)" == "0" && $(basename "$1") == "elasticsearch" ]]; then
# centos:7 chroot doesn't have the --skip-chdir option and
# changes our CWD.
# Rewrite CMD args to replace $1 with elasticsearch explicitly,
#so that we are backwards compatible with the docs
#from the previous Elasticsearch versions<6
#and configuration option D:

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#_d_override_the_image_8217_s_default_ulink_url_https_docs_docker_com_engine_reference_run_cmd_default_command_or_options_cmd_ulink

# Without this, user could specify `elasticsearch -E x.y=z` but
# `bin/elasticsearch -E x.y=z` would not work.
set -- "elasticsearch" "${@:2}"
# Use chroot to switch to UID 1000
exec chroot --userspec=1000 / "$@"

else
# User probably wants to run something else, like /bin/bash, with another uid forced (Openshift?)
exec "$@"
fi
fi

#Parse Docker env vars to customize Elasticsearch
#e.g. Setting the env var cluster.name=testcluster
#will cause Elasticsearch to be invoked with -Ecluster.name=testcluster
#see https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#_setting_default_settings

declare -a es_opts

while IFS='=' read -r envvar_key envvar_value
do
#Elasticsearch settings need to have at least two dot separated lowercase
#words, e.g. cluster.name, except for processors which we handle
#specially
if [[ "$envvar_key" =~ ^[a-z0-9_]+.[a-z0-9_]+ || "$envvar_key" == "processors" ]]; then
if [[ ! -z envvar_value ]]; then es_opt="-E{envvar_key}={envvar_value}" es_opts+=("{es_opt}")
fi
fi
done < <(env)

#The virtual file /proc/self/cgroup should list the current cgroup
#membership. For each hierarchy, you can follow the cgroup path from
#this file to the cgroup filesystem (usually /sys/fs/cgroup/) and
#introspect the statistics for the cgroup for the given
#hierarchy. Alas, Docker breaks this by mounting the container
#statistics at the root while leaving the cgroup paths as the actual
#paths. Therefore, Elasticsearch provides a mechanism to override
#reading the cgroup path from /proc/self/cgroup and instead uses the
#cgroup path defined the JVM system property
#es.cgroups.hierarchy.override. Therefore, we set this value here so
#that cgroup statistics are available for the container this process
#will run in.
export ES_JAVA_OPTS="-Des.cgroups.hierarchy.override=/ $ES_JAVA_OPTS"

if [[ -f bin/elasticsearch-users ]]; then
#Check for the ELASTIC_PASSWORD environment variable to set the
#bootstrap password for Security.

#This is only required for the first node in a cluster with Security
#enabled, but we have no way of knowing which node we are yet. We'll just
#honor the variable if it's present.
if [[ -n "ELASTIC_PASSWORD" ]]; then [[ -f /usr/share/elasticsearch/config/elasticsearch.keystore ]] || (run_as_other_user_if_needed elasticsearch-keystore create) if ! (run_as_other_user_if_needed elasticsearch-keystore list | grep -q '^bootstrap.password'); then
(run_as_other_user_if_needed echo "$ELASTIC_PASSWORD" | elasticsearch-keystore add -x 'bootstrap.password')
fi
else
/usr/share/elasticsearch/bin/elasticsearch-keystore upgrade
fi
fi

if [[ "$(id -u)" == "0" ]]; then
#If requested and running as root, mutate the ownership of bind-mounts
if [[ -n "$TAKE_FILE_OWNERSHIP" ]]; then
chown -R 1000:0 /usr/share/elasticsearch/{data,logs}
fi
fi

run_as_other_user_if_needed /usr/share/elasticsearch/bin/elasticsearch "${es_opts[@]}"

then I tried to change the docker-entrypoint.sh with below and it also failing with the error message
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.2.jar:7.9.2]

github.com

elastic/elasticsearch/blob/master/distribution/docker/src/docker/bin/docker-entrypoint.sh

#!/bin/bash
set -e

# Files created by Elasticsearch should always be group writable too
umask 0002

# Allow user specify custom CMD, maybe bin/elasticsearch itself
# for example to directly specify `-E` style parameters for elasticsearch on k8s
# or simply to run /bin/bash to check the image
if [[ "$1" == "eswrapper" || $(basename "$1") == "elasticsearch" ]]; then
  # Rewrite CMD args to remove the explicit command,
  # so that we are backwards compatible with the docs
  # from the previous Elasticsearch versions < 6
  # and configuration option:
  # https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html#_d_override_the_image_8217_s_default_ulink_url_https_docs_docker_com_engine_reference_run_cmd_default_command_or_options_cmd_ulink
  # Without this, user could specify `elasticsearch -E x.y=z` but
  # `bin/elasticsearch -E x.y=z` would not work. In any case,
  # we want to continue through this script, and not exec early.
  set -- "${@:2}"
else

This file has been truncated. show original

can you please help me on this.

sebgl · December 23, 2020, 9:56am

@NK2812 running Elasticsearch with ECK strictly requires xpack.security.enabled: true (which is set by ECK).
There is no way around this.

Christian_Dahlqvist · December 29, 2020, 10:25am

The operator requires the default distribution with security enabled and will not work with third party security plugins.

system · January 26, 2021, 10:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.