Elasticsearch 5.0.1 docker official image failing to start

Elastic Stack Elasticsearch
1

Good evening,

I'm trying to start elasticsearch 5.0.1 docker official image using following command line:

docker -D -H unix:///var/run/docker.sock run --cpu-shares 1024 --memory 536870912 -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /system/mesos/elasticsearchf/:/data/:rw -v /system/mesos/elasticsearchf/config:/config/:rw --net bridge -p 9200:9200/tcp -p 9300:9300/tcp elasticsearch

DEBU[0008] Corrupted prefix: []
DEBU[0008] [hijack] End of stdout

But if I run following command:

docker -D -H unix:///var/run/docker.sock run -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /system/mesos/elasticsearchf/:/data/:rw -v /system/mesos/elasticsearchf/config:/config/:rw --net bridge -p 9200:9200/tcp -p 9300:9300/tcp elasticsearch DEBU[0010] framesize: 80

[2016-11-23T14:51:42,650][INFO ][o.e.n.Node ] [] initializing ...
DEBU[0010] framesize: 374
[2016-11-23T14:51:42,740][INFO ][o.e.e.NodeEnvironment ] [JH06BUU] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/centos-root)]], net usable_space [33gb], net total_space [37.4gb], spins? [possibly], types [xfs]
[2016-11-23T14:51:42,741][INFO ][o.e.e.NodeEnvironment ] [JH06BUU] heap size [1.9gb], compressed ordinary object pointers [true]
DEBU[0010] framesize: 140
[2016-11-23T14:51:42,742][INFO ][o.e.n.Node ] [JH06BUU] node name [JH06BUU] derived from node ID; set [node.name] to override
DEBU[0010] framesize: 249
[2016-11-23T14:51:42,744][INFO ][o.e.n.Node ] [JH06BUU] version[5.0.1], pid[1], build[080bb47/2016-11-11T22:08:49.812Z], OS[Linux/3.10.0-327.36.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_111/25.111-b14]
DEBU[0011] framesize: 306

it works properly.

Configuration file content

cat elasticsearch.yml

path:
data: /data
logs: /logs
shared_data: /data

node.add_lock_id_to_custom_path: false
index.number_of_shards:2
cluster.name: elasticcluster
discovery.zen.ping.unicast.hosts: ["10.120.1.190", "10.120.1.191"]

Can you help me to understand why is this happening?

Kind Regards

2

Hello,

The issue is that you are not using the Docker image maintained by Elastic (docker.elastic.co/elasticsearch/elasticsearch:5.0.1) but rather an Elasticsearch image that is not maintained by Elastic hosted in the Docker Hub Registry.

Please refer to our instructions on how to use the image: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/docker.html

Thank you

3

Good morning Dimitrios,

Thanks for your answer.

I've retested with the recommended image and results are the same.

docker -D -H unix:///var/run/docker.sock run --cpu-shares 1024 --memory 136870912 -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /system/mesos/elasticsearchf/:/data/:rw -v /system/mesos/elasticsearchf/config:/config/:rw --net bridge -p 9200:9200/tcp -p 9300:9300/tcp docker.elastic.co/elasticsearch/elasticsearch:5.0.1

DEBU[0004] Corrupted prefix: []
DEBU[0004] [hijack] End of stdout

docker -D -H unix:///var/run/docker.sock run -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /system/mesos/elasticsearchf/:/data/:rw -v /system/mesos/elasticsearchf/config:/config/:rw --net bridge -p 9200:9200/tcp -p 9300:9300/tcp docker.elastic.co/elasticsearch/elasticsearch:5.0.1

DEBU[0009] framesize: 80
[2016-11-28T09:53:48,931][INFO ][o.e.n.Node ] [] initializing ...
DEBU[0009] framesize: 199
[2016-11-28T09:53:49,545][INFO ][o.e.e.NodeEnvironment ] [EGDfRUK] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [9.7gb], net total_space [9.9gb], spins? [unknown], types [rootfs]
DEBU[0009] framesize: 132
[2016-11-28T09:53:49,545][INFO ][o.e.e.NodeEnvironment ] [EGDfRUK] heap size [1.9gb], compressed ordinary object pointers [true]

Regards

4

Hello,

I am not sure what is the memory limit you are trying to set. The docker run reference mentions the units b, k, m, g.

If I run the image using something like

docker run --rm --cpu-shares 1024 --memory 2G -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:5.0.1

it works fine here, but setting memory to the value you have 136870912 (130MB?) does not work and anyway would be too little for ES.

Please try instead setting the memory using the Docker recommendations for the units.

Additionally, note that the data dir for the image is /usr/share/elasticsearch/data as opposed to data; same holds for '/config, the correct conf dir is/usr/share/elasticsearch/config`. Please refer to the official documentation that describes the paths.

Best Regards

5

Good morning Dimitrios,

Thanks for your help on this topic.

I've followed instructions you detailed and followed

    https://www.elastic.co/guide/en/elasticsearch/reference/5.0/settings.html

in order to configure elasticsearch.yml and log4j2.properties files, those files are located at:

pwd

    /system/mesos/elasticsearchf/config

ls

     elasticsearch.yml  log4j2.properties

Contain of these configuration files:

cat elasticsearch.yml

path:
data: /usr/share/elasticsearch/data
logs: /usr/share/elasticsearch/data
shared_data: /usr/share/elasticsearch/data
config: /usr/share/elasticsearch/config

node.add_lock_id_to_custom_path: false
index.number_of_shards: 2
cluster.name: elasticcluster
discovery.zen.ping.unicast.hosts: ["10.120.1.190", "10.120.1.191"]

cat log4j2.properties

appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = {sys:es.logs}.log appender.rolling.layout.type = PatternLayout appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %.10000m%n appender.rolling.filePattern = {sys:es.logs}-%d{yyyy-MM-dd}.log
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true

And when I run command:

# docker -D -H unix:///var/run/docker.sock run --cpu-shares 1024 --memory 2634022912 -e ES_JAVA_OPTS="-Xms2G -Xmx2G" -v /system/mesos/elasticsearchf/data:/usr/share/elasticsearch/data/:rw -v /system/mesos/elasticsearchf/config:/usr/share/elasticsearch/config/:rw --net bridge -p 9200:9200/tcp -p 9300:9300/tcp docker.elastic.co/elasticsearch/elasticsearch:5.0.1

I got error message shown in attached file

any idea why this is happening

Thanks in advance for your time

Kind Regards

elasticsearch_docker_failure_1.PNG1245×529 18.8 KB

elasticsearch_docker_failure_2.PNG1101×478 14.3 KB

6

@cruizpollino I've managed to reproduce this if there are any spaces after RollingFile in the line

appender.rolling.type = RollingFile

Can you please confirm whether or not there are such spaces at the end of any of the lines in your log4j2.properties, and if the issue stops after removing all such spaces? Otherwise, I am not able to reproduce this with exactly the following content in log4j2.properties:

status = error
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs}.log
appender.rolling.layout.type = PatternLayout
appender.rolling.layout.pattern = [%d{ISO8601}][%-5p][%-25c] %.10000m%n
appender.rolling.filePattern = ${sys:es.logs}-%d{yyyy-MM-dd}.log
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
rootLogger.level = info
rootLogger.appenderRef.rolling.ref = rolling

with no spaces at the end of any of the lines.

7

Good morning Jason,

sorry for late reply.

You were right there were a space after some of the lines and rolling.type was one of those.

I fixed those lines and now the error is not appearing anymore, now the docker execution commands simply fails with now error message apart from Corrupted prefix and end of stdout lines.

in order to troubleshoot the issue I set docker log-level to debug and run "tail -f /var/log/messages" at same time I run the docker command.

Attached images contain the output during that time period

any idea is really welcome

Kind Regards

elasticsearch_docker_failure_1_Dec7th.PNG1255×891 68.7 KB

elasticsearch_docker_failure_2_Dec7th.PNG966×885 62.4 KB

elasticsearch_docker_failure_3_Dec7th.PNG1242×853 55 KB

elasticsearch_docker_failure_4_Dec7th.PNG1237×898 60.5 KB

elasticsearch_docker_failure_5_Dec7th.PNG1200×544 41.3 KB

8

in addition to my previous message, i've tried to run

$ docker -D -H unix:///var/run/docker.sock run --cpu-shares 1024 --memory 2634022912 -e ES_JAVA_OPTS="-Xms2G -Xmx2G" --net bridge -p 9200:9200/tcp -p 9300:9300/tcp docker.elastic.co/elasticsearch/elasticsearch:5.0.1

and it starts properly, so looks like something in the config files is breaking the start process, but don't know where to look to find what

Regards

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.