Elasticsearch on CentOS 6.9

Quadrosh · October 27, 2017, 2:46pm

Hello, i have problem with install elasticsearch on CentOS 6.9
install itself goes well (by creating a file elasticsearch.repo in the /etc/yum.repos.d/ and run sudo yum install elasticsearch)

but command "sudo -i service elasticsearch start" return "[FAILED]"

here the output of "set -x"

'[' -f /etc/rc.status ']'
'[' -f /etc/rc.d/init.d/functions ']'
. /etc/rc.d/init.d/functions

++ TEXTDOMAIN=initscripts

++ umask 022

++ PATH=/sbin:/usr/sbin:/bin:/usr/bin

++ export PATH

++ '[' -z '' ']'

++ COLUMNS=80

++ '[' -z '' ']'

+++ /sbin/consoletype

++ CONSOLETYPE=pty

++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'

++ . /etc/profile.d/lang.sh

++ unset LANGSH_SOURCED

++ '[' -z '' ']'

++ '[' -f /etc/sysconfig/init ']'

++ . /etc/sysconfig/init

+++ BOOTUP=color

+++ RES_COL=60

+++ MOVE_TO_COL='echo -en \033[60G'

+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'

+++ SETCOLOR_FAILURE='echo -en \033[0;31m'

+++ SETCOLOR_WARNING='echo -en \033[0;33m'

+++ SETCOLOR_NORMAL='echo -en \033[0;39m'

+++ PROMPT=no

+++ AUTOSWAP=no

+++ ACTIVE_CONSOLES='/dev/tty[1-6]'

+++ SINGLE=/sbin/sushell

++ '[' pty = serial ']'

+++ cat /proc/cmdline

++ strstr quiet rc.debug

++ '[' quiet = quiet ']'

++ return 1

ES_USER=elasticsearch
ES_GROUP=elasticsearch
ES_HOME=/usr/share/elasticsearch
MAX_OPEN_FILES=65536
MAX_MAP_COUNT=262144
LOG_DIR=/var/log/elasticsearch
DATA_DIR=/var/lib/elasticsearch
CONF_DIR=/etc/elasticsearch
PID_DIR=/var/run/elasticsearch
ES_ENV_FILE=/etc/sysconfig/elasticsearch
'[' -f /etc/sysconfig/elasticsearch ']'
. /etc/sysconfig/elasticsearch

++ ES_STARTUP_SLEEP_TIME=5

'[' elasticsearch '!=' elasticsearch ']'
'[' elasticsearch '!=' elasticsearch ']'
'[' '!' -z '' ']'
exec=/usr/share/elasticsearch/bin/elasticsearch
prog=elasticsearch
pidfile=/var/run/elasticsearch/elasticsearch.pid
export ES_JAVA_OPTS
export JAVA_HOME
export ES_INCLUDE
export ES_JVM_OPTIONS
export ES_STARTUP_SLEEP_TIME
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
lockfile=/var/lock/subsys/elasticsearch
'[' -n ']'
'[' -z elasticsearch ']'
'[' '!' -x /usr/share/elasticsearch/bin/elasticsearch ']'
case "$1" in
rh_status_q
rh_status
start
checkJava
'[' -x /bin/java ']'

++ which java

JAVA=/usr/bin/java
'[' '!' -x /usr/bin/java ']'
'[' -x /usr/share/elasticsearch/bin/elasticsearch ']'
'[' -n 65536 ']'
ulimit -n 65536
'[' -n '' ']'
'[' -n 262144 -a -f /proc/sys/vm/max_map_count ']'
sysctl -q -w vm.max_map_count=262144

error: permission denied on key 'vm.max_map_count'

'[' -n /var/run/elasticsearch ']'
'[' '!' -e /var/run/elasticsearch ']'
'[' -n /var/run/elasticsearch/elasticsearch.pid ']'
'[' '!' -e /var/run/elasticsearch/elasticsearch.pid ']'
cd /usr/share/elasticsearch
echo -n 'Starting elasticsearch: '

Starting elasticsearch: + daemon --user elasticsearch --pidfile /var/run/elasticsearch/elasticsearch.pid /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

local gotbase= force= nicelevel corelimit
local pid base= user= nice= bg= pid_file=
local cgroup=
nicelevel=0
'[' --user '!=' -user ']'
case $1 in
user=elasticsearch
shift 2
'[' --pidfile '!=' -pidfile ']'
case $1 in
pid_file=/var/run/elasticsearch/elasticsearch.pid
shift 2
'[' /usr/share/elasticsearch/bin/elasticsearch '!=' /usr/share/elasticsearch/bin/elasticsearch ']'
'[' -z '' ']'
base=elasticsearch
__pids_var_run elasticsearch /var/run/elasticsearch/elasticsearch.pid
local base=elasticsearch
local pid_file=/var/run/elasticsearch/elasticsearch.pid

++ /usr/bin/dirname /var/run/elasticsearch/elasticsearch.pid

local pid_dir=/var/run/elasticsearch
local binary=
'[' -d /var/run/elasticsearch -a '!' -r /var/run/elasticsearch ']'
pid=
'[' -f /var/run/elasticsearch/elasticsearch.pid ']'
local line p
'[' '!' -r /var/run/elasticsearch/elasticsearch.pid ']'
:
read line
'[' -z '' ']'
break
'[' -n '' ']'
return 1
'[' -n '' -a -z '' ']'
corelimit='ulimit -S -c 0'
'[' -n '' ']'
'[' -n '' ']'
'[' color = verbose -a -z '' ']'
'[' -z elasticsearch ']'
runuser -s /bin/bash elasticsearch -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch'

Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

/usr/share/elasticsearch/bin/elasticsearch: line 205: 13375 Killed exec "$JAVA" $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch "$@" 0>&-

'[' 1 -eq 0 ']'
failure 'elasticsearch startup'
local rc=1
'[' color '!=' verbose -a -z '' ']'
echo_failure
'[' color = color ']'

echo -en '\033[60G'

                                                     + echo -n '['

[+ '[' color = color ']'

echo -en '\033[0;31m'
echo -n FAILED

FAILED+ '[' color = color ']'

echo -en '\033[0;39m'
echo -n ']'

]+ echo -ne '\r'

return 1
'[' -x /bin/plymouth ']'
/bin/plymouth --details
return 1
retval=1
echo
'[' 1 -eq 0 ']'
return 1
exit 1

On this server installed java - java version "9.0.1"

Java(TM) SE Runtime Environment (build 9.0.1+11)

Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

Could you please steer me in the right direction

pjanzen · October 27, 2017, 2:53pm

Elastic is not (yet) supported on java 9, please install java 8.

Quadrosh · October 27, 2017, 4:22pm

Thanks for your answer, i removed java 9 and installed java 8 from Oracle (jdk-8u144-linux-x64.rpm), reboot server, and elastic start still failed.
Here the out of set -x option - as i see it's the same

'[' -f /etc/rc.status ']'
'[' -f /etc/rc.d/init.d/functions ']'
. /etc/rc.d/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=no
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/(~|.bak|.orig|.rpmnew|.rpmorig|.rpmsave)$/d'
+++ cat /proc/cmdline
++ strstr quiet rc.debug
++ '[' quiet = quiet ']'
++ return 1
ES_USER=elasticsearch
ES_GROUP=elasticsearch
ES_HOME=/usr/share/elasticsearch
MAX_OPEN_FILES=65536
MAX_MAP_COUNT=262144
LOG_DIR=/var/log/elasticsearch
DATA_DIR=/var/lib/elasticsearch
CONF_DIR=/etc/elasticsearch
PID_DIR=/var/run/elasticsearch
ES_ENV_FILE=/etc/sysconfig/elasticsearch
'[' -f /etc/sysconfig/elasticsearch ']'
. /etc/sysconfig/elasticsearch
++ ES_STARTUP_SLEEP_TIME=5
'[' elasticsearch '!=' elasticsearch ']'
'[' elasticsearch '!=' elasticsearch ']'
'[' '!' -z '' ']'
exec=/usr/share/elasticsearch/bin/elasticsearch
prog=elasticsearch
pidfile=/var/run/elasticsearch/elasticsearch.pid
export ES_JAVA_OPTS
export JAVA_HOME
export ES_INCLUDE
export ES_JVM_OPTIONS
export ES_STARTUP_SLEEP_TIME
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
test -n ''
lockfile=/var/lock/subsys/elasticsearch
'[' -n ']'
'[' -z elasticsearch ']'
'[' '!' -x /usr/share/elasticsearch/bin/elasticsearch ']'
case "$1" in
rh_status_q
rh_status
start
checkJava
'[' -x /bin/java ']'
++ which java
JAVA=/usr/bin/java
'[' '!' -x /usr/bin/java ']'
'[' -x /usr/share/elasticsearch/bin/elasticsearch ']'
'[' -n 65536 ']'
ulimit -n 65536
'[' -n '' ']'
'[' -n 262144 -a -f /proc/sys/vm/max_map_count ']'
sysctl -q -w vm.max_map_count=262144
error: permission denied on key 'vm.max_map_count'
'[' -n /var/run/elasticsearch ']'
'[' '!' -e /var/run/elasticsearch ']'
'[' -n /var/run/elasticsearch/elasticsearch.pid ']'
'[' '!' -e /var/run/elasticsearch/elasticsearch.pid ']'
cd /usr/share/elasticsearch
echo -n 'Starting elasticsearch: '
Starting elasticsearch: + daemon --user elasticsearch --pidfile /var/run/elasticsearch/elasticsearch.pid /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch
local gotbase= force= nicelevel corelimit
local pid base= user= nice= bg= pid_file=
local cgroup=
nicelevel=0
'[' --user '!=' -user ']'
case $1 in
user=elasticsearch
shift 2
'[' --pidfile '!=' -pidfile ']'
case $1 in
pid_file=/var/run/elasticsearch/elasticsearch.pid
shift 2
'[' /usr/share/elasticsearch/bin/elasticsearch '!=' /usr/share/elasticsearch/bin/elasticsearch ']'
'[' -z '' ']'
base=elasticsearch
__pids_var_run elasticsearch /var/run/elasticsearch/elasticsearch.pid
local base=elasticsearch
local pid_file=/var/run/elasticsearch/elasticsearch.pid
++ /usr/bin/dirname /var/run/elasticsearch/elasticsearch.pid
local pid_dir=/var/run/elasticsearch
local binary=
'[' -d /var/run/elasticsearch -a '!' -r /var/run/elasticsearch ']'
pid=
'[' -f /var/run/elasticsearch/elasticsearch.pid ']'
local line p
'[' '!' -r /var/run/elasticsearch/elasticsearch.pid ']'
:
read line
'[' -z '' ']'
break
'[' -n '' ']'
return 1
'[' -n '' -a -z '' ']'
corelimit='ulimit -S -c 0'
'[' -n '' ']'
'[' -n '' ']'
'[' color = verbose -a -z '' ']'
'[' -z elasticsearch ']'
runuser -s /bin/bash elasticsearch -c 'ulimit -S -c 0 >/dev/null 2>&1 ; /usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch'
/usr/share/elasticsearch/bin/elasticsearch: line 205: 2459 Killed exec "$JAVA" $ES_JAVA_OPTS -Des.path.home="$ES_HOME" -cp "$ES_CLASSPATH" org.elasticsearch.bootstrap.Elasticsearch "$@" 0>&-
'[' 1 -eq 0 ']'
failure 'elasticsearch startup'
local rc=1
'[' color '!=' verbose -a -z '' ']'
echo_failure
'[' color = color ']'
echo -en '\033[60G'
+ echo -n '['
[+ '[' color = color ']'
echo -en '\033[0;31m'
echo -n FAILED
FAILED+ '[' color = color ']'
echo -en '\033[0;39m'
echo -n ']'
]+ echo -ne '\r'
return 1
'[' -x /bin/plymouth ']'
/bin/plymouth --details
return 1
retval=1
echo
'[' 1 -eq 0 ']'
return 1
exit 1

pjanzen · October 27, 2017, 5:03pm

And there is nothing in the logs?

Quadrosh · October 27, 2017, 5:40pm

could you please tell me, where i can find the logs?

pjanzen · October 27, 2017, 5:58pm

/var/log/elasticsearch ??

Quadrosh · October 27, 2017, 6:03pm

it's empty

pjanzen · October 27, 2017, 6:12pm

What happens when you start it on the CLI ?

/usr/share/elasticsearch/bin/elasticsearch -p /var/run/elasticsearch/elasticsearch.pid -d -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

Quadrosh · October 27, 2017, 6:24pm

nothing happened, empty out

pjanzen · October 27, 2017, 6:26pm

Ok, please post output of.

java -version
cat /etc/elasticsearch/elasticsearch.yml
cat /etc/elasticsearch/jvm.options
mem size and total cpus in the box / container / vmware image

Quadrosh · October 27, 2017, 6:43pm

java -version
java version "1.8.0_144"
Java(TM) SE Runtime Environment (build 1.8.0_144-b01)
Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)

cat /etc/elasticsearch/elasticsearch.yml

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

#path.data: /path/to/data

Path to log files:

#path.logs: /path/to/logs

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: 192.168.0.1

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.zen.ping.unicast.hosts: ["host1", "host2"]

Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

#discovery.zen.minimum_master_nodes: 3

For more information, consult the zen discovery module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

cat /etc/elasticsearch/jvm.options

JVM configuration

################################################################

IMPORTANT: JVM heap size

################################################################

You should always set the min and max JVM heap

size to the same value. For example, to set

the heap to 4 GB, set:

-Xms4g

-Xmx4g

See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

for more information

################################################################

Xms represents the initial size of total heap space

Xmx represents the maximum size of total heap space

-Xms2g
-Xmx2g

################################################################

Expert settings

################################################################

All settings below this section are considered

expert settings. Don't tamper with them unless

you understand what you are doing

################################################################

GC configuration

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly

optimizations

pre-touch memory pages used by the JVM during initialization

-XX:+AlwaysPreTouch

basic

force the server VM (remove on 32-bit client JVMs)

-server

explicitly set the stack size (reduce to 320k on 32-bit client JVMs)

-Xss1m

set to headless, just in case

-Djava.awt.headless=true

ensure UTF-8 encoding by default (e.g. filenames)

-Dfile.encoding=UTF-8

use our provided JNA always versus the system one

-Djna.nosys=true

use old-style file permissions on JDK9

-Djdk.io.permissionsUseCanonicalPath=true

flags to configure Netty

-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0

log4j 2

-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true

heap dumps

generate a heap dump when an allocation from the Java heap fails

heap dumps are created in the working directory of the JVM

-XX:+HeapDumpOnOutOfMemoryError

specify an alternative path for heap dumps

ensure the directory exists and has sufficient space

#-XX:HeapDumpPath=${heap.dump.path}

GC logging

#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime

log GC status to a file with time stamps

ensure the directory exists

#-Xloggc:${loggc}

By default, the GC log file will not rotate.

By uncommenting the lines below, the GC log file

will be rotated every 128MB at most 32 times.

#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M

Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.

If documents were already indexed with unquoted fields in a previous version

of Elasticsearch, some operations may throw errors.

WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided

only for migration purposes.

#-Delasticsearch.json.allow_unquoted_field_names=true

mem size and total cpus in the box / container / vmware image
-bash: mem: command not found

pjanzen · October 27, 2017, 6:50pm

Ok I do not see any strange things here.. In regard to mem .. this is not a command but merely a question as in how much memory is installed in your box or configured for your installation.

that said I cannot help you any further as I have no clue why your commands are not working..

Quadrosh · October 27, 2017, 6:53pm

ok, maybe it helps..

cat /proc/meminfo
MemTotal: 1048576 kB
MemFree: 703768 kB
Cached: 170840 kB
Buffers: 0 kB
Active: 135960 kB
Inactive: 168384 kB
Active(anon): 77268 kB
Inactive(anon): 61060 kB
Active(file): 58692 kB
Inactive(file): 107324 kB
Unevictable: 32 kB
Mlocked: 32 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 32 kB
Writeback: 0 kB
AnonPages: 138328 kB
Shmem: 4824 kB
Slab: 40428 kB
SReclaimable: 12156 kB
SUnreclaim: 28272 kB

pjanzen · October 27, 2017, 6:58pm

you set:
-Xms2g
-Xmx2g

but you only have 1G mem in the box / container..

Quadrosh · October 27, 2017, 7:08pm

How can i change it to the right value? And which value is right in my case?

pjanzen · October 27, 2017, 7:10pm

well vi the jvm.options file and adjust it. I would recommend to add memory to your machine. say added 3G then you dont have to change the file.

Quadrosh · October 27, 2017, 7:34pm

Thank you for your help.
I adjusted the value and elastic runs now.
I will add the memory asap)

warkolm · October 29, 2017, 8:14am

In future please format any CLI/shell output using the </> button, it makes it much easier to read.

system · November 26, 2017, 8:14am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.