Elastic Search with Docker

nages · August 21, 2020, 6:13pm

I ran the following docker compose script and I am expecting two nodes to be up, however this only one. There seems to be some obvious error , please suggest your inputs

Taken from the documentation https://www.elastic.co/guide/en/elasticsearch/reference/6.8/docker.html

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.12
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
    networks:
      - esnet
  elasticsearch2:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.12
    container_name: elasticsearch2
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - "discovery.zen.ping.unicast.hosts=elasticsearch"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata2:/usr/share/elasticsearch/data
    networks:
      - esnet

volumes:
  esdata1:
    driver: local
  esdata2:
    driver: local

networks:
  esnet:

http://127.0.0.1:9200/_cat/health

1598033352 18:09:12 docker-cluster green 1 1 0 0 0 0 0 0 - 100.0%

docker-compose ps

Name                   Command               State                Ports              
------------------------------------------------------------------------------------------
elasticsearch    /usr/local/bin/docker-entr ...   Up      0.0.0.0:9200->9200/tcp, 9300/tcp
elasticsearch2   /usr/local/bin/docker-entr ...   Up      9200/tcp, 9300/tcp

Emanuil · August 21, 2020, 6:37pm

Hmm, interesting. Did you wait long enough for them to find each other and form a cluster post boot-up? The docker-compose.yml you've pasted and the example one in the docs are exactly the same. When I run it, I get

1598034943 18:35:43 docker-cluster green 2 2 0 0 0 0 0 0 - 100.0%

i.e. 2 nodes of this exact version of Elasticsearch, exactly as it's supposed to work. I can't see an obvious reason why I can't reproduce it.

nages · August 21, 2020, 6:59pm

hmm , no luck.
when did you say "wait long enough" - how long do you mean ? 1 min ?

Emanuil · August 21, 2020, 7:06pm

Yeah, 1 min should be enough, but depends on your hardware. Usually it takes 20-30 seconds. I guess if it's not joined after 3 minutes you really have a problem. Here are my logs from the same example you're trying:

elasticsearch     | [2020-08-21T18:34:18,866][INFO ][o.e.c.s.MasterService    ] [DsbxDD4] zen-disco-node-join[{pfFjLH4}{pfFjLH4BTMulc2Mi6ilmPA}{mvCbPYD3SY2y_43qNIGYYw}{172.18.0.3}{172.18.0.3:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason: added {{pfFjLH4}{pfFjLH4BTMulc2Mi6ilmPA}{mvCbPYD3SY2y_43qNIGYYw}{172.18.0.3}{172.18.0.3:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
elasticsearch2    | [2020-08-21T18:34:18,912][INFO ][o.e.c.s.ClusterApplierService] [pfFjLH4] detected_master {DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [12]])
elasticsearch2    | [2020-08-21T18:34:18,971][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [pfFjLH4] Failed to clear cache for realms [[]]
elasticsearch2    | [2020-08-21T18:34:18,975][INFO ][o.e.x.s.a.TokenService   ] [pfFjLH4] refresh keys
elasticsearch2    | [2020-08-21T18:34:19,109][INFO ][o.e.x.s.a.TokenService   ] [pfFjLH4] refreshed keys
elasticsearch     | [2020-08-21T18:34:19,130][INFO ][o.e.c.s.ClusterApplierService] [DsbxDD4] added {{pfFjLH4}{pfFjLH4BTMulc2Mi6ilmPA}{mvCbPYD3SY2y_43qNIGYYw}{172.18.0.3}{172.18.0.3:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [12] source [zen-disco-node-join[{pfFjLH4}{pfFjLH4BTMulc2Mi6ilmPA}{mvCbPYD3SY2y_43qNIGYYw}{172.18.0.3}{172.18.0.3:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]]])
elasticsearch     | [2020-08-21T18:34:19,141][WARN ][o.e.d.z.ElectMasterService] [DsbxDD4] value for setting "discovery.zen.minimum_master_nodes" is too low. This can result in data loss! Please set it to at least a quorum of master-eligible nodes (current value: [-1], total number of master-eligible nodes used for publishing in this round: [2])
elasticsearch2    | [2020-08-21T18:34:19,160][INFO ][o.e.h.n.Netty4HttpServerTransport] [pfFjLH4] publish_address {172.18.0.3:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch2    | [2020-08-21T18:34:19,161][INFO ][o.e.n.Node               ] [pfFjLH4] started
elasticsearch2    | [2020-08-21T18:34:19,267][INFO ][o.e.l.LicenseService     ] [pfFjLH4] license [29417248-2d03-4cbd-a49e-4201736499da] mode [basic] - valid
elasticsearch2    | [2020-08-21T18:34:19,274][INFO ][o.e.x.m.e.l.LocalExporter] [pfFjLH4] waiting for elected master node [{DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
elasticsearch     | [2020-08-21T18:34:19,321][INFO ][o.e.l.LicenseService     ] [DsbxDD4] license [29417248-2d03-4cbd-a49e-4201736499da] mode [basic] - valid
elasticsearch2    | [2020-08-21T18:34:19,376][INFO ][o.e.x.m.e.l.LocalExporter] [pfFjLH4] waiting for elected master node [{DsbxDD4}{DsbxDD4lQ9-V7YVZ7RQ9kg}{h-TMjWdyRjGqRn_M9VMUfw}{172.18.0.2}{172.18.0.2:9300}{ml.machine_memory=4129218560, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

See especially zen-disco-node-join and waiting for elected master messages.

Try doing the docker-compose up startup and paste your entire log here, let's see what's going wrong. If it won't fit in a forum post, use https://gist.github.com/ .

nages · August 21, 2020, 7:23pm

Here you go

gist.github.com

https://gist.github.com/nag9s/5bf6f2f8d73672fc95b56aa07e845442

docker-compose.yml

version: '2.2'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.8.12
    container_name: elasticsearch
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:

This file has been truncated. show original

log file

Attaching to elasticsearch3, elasticsearch, kibana, elasticsearch2
[36melasticsearch     |[0m OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1
[36melasticsearch     |[0m [2020-08-21T19:09:24,657][INFO ][o.e.e.NodeEnvironment    ] [aJz-wqt] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [13.4gb], net total_space [21.4gb], types [ext4]
[36melasticsearch     |[0m [2020-08-21T19:09:24,755][INFO ][o.e.e.NodeEnvironment    ] [aJz-wqt] heap size [512mb], compressed ordinary object pointers [true]
[36melasticsearch     |[0m [2020-08-21T19:09:24,806][INFO ][o.e.n.Node               ] [aJz-wqt] node name derived from node ID [aJz-wqtGQ9iUpjVxNEXW4g]; set [node.name] to override
[36melasticsearch     |[0m [2020-08-21T19:09:24,831][INFO ][o.e.n.Node               ] [aJz-wqt] version[6.8.12], pid[1], build[default/docker/7a15d2a/2020-08-12T07:27:20.804867Z], OS[Linux/5.4.0-42-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/14.0.1/14.0.1+7]
[36melasticsearch     |[0m [2020-08-21T19:09:24,832][INFO ][o.e.n.Node               ] [aJz-wqt] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-802910741859875890, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
[36melasticsearch     |[0m [2020-08-21T19:10:03,279][INFO ][o.e.p.PluginsService     ] [aJz-wqt] loaded module [aggs-matrix-stats]
[36melasticsearch     |[0m [2020-08-21T19:10:03,291][INFO ][o.e.p.PluginsService     ] [aJz-wqt] loaded module [analysis-common]
[36melasticsearch     |[0m [2020-08-21T19:10:03,291][INFO ][o.e.p.PluginsService     ] [aJz-wqt] loaded module [ingest-common]

This file has been truncated. show original

I also happened to add one more service Elasticsearch service and kibana. Hope it should not be a problem

Emanuil · August 21, 2020, 7:31pm

What operating system and docker versions are you running this under? I'm curious since I wonder if something is interfering with the esnet docker network and the nodes are somehow not seeing each other.

nages · August 21, 2020, 7:34pm

ubuntu
i tool feel the same, it seems both are NOT aware of each other and creating their own clusters?

nages · August 21, 2020, 7:38pm

hmm, this confirms that - both are trying to compete each other to become master nodes ?

> elasticsearch2 |e[0m [2020-08-21T19:28:22,088][INFO ][o.e.c.s.MasterService ] [MJxcYj2] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {MJxcYj2}{MJxcYj2aQB2sVhBGDGQ7Pg}{Ejr7_xXSSa2LC-DHa5vHdg}{172.25.0.4}{172.25.0.4:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} > e[33melasticsearch2 |e[0m [2020-08-21T19:28:22,157][INFO ][o.e.c.s.ClusterApplierService] [MJxcYj2] new_master {MJxcYj2}{MJxcYj2aQB2sVhBGDGQ7Pg}{Ejr7_xXSSa2LC-DHa5vHdg}{172.25.0.4}{172.25.0.4:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {MJxcYj2}{MJxcYj2aQB2sVhBGDGQ7Pg}{Ejr7_xXSSa2LC-DHa5vHdg}{172.25.0.4}{172.25.0.4:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) > elasticsearch |e[0m [2020-08-21T19:28:22,296][INFO ][o.e.c.s.MasterService ] [xwrHIvk] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {xwrHIvk}{xwrHIvkxRQmSnETYdQsWUQ}{Ull3Yz5HTOWwGeCOnOBApA}{172.25.0.3}{172.25.0.3:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} > e[36melasticsearch |e[0m [2020-08-21T19:28:22,367][INFO ][o.e.c.s.ClusterApplierService] [xwrHIvk] new_master {xwrHIvk}{xwrHIvkxRQmSnETYdQsWUQ}{Ull3Yz5HTOWwGeCOnOBApA}{172.25.0.3}{172.25.0.3:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {xwrHIvk}{xwrHIvkxRQmSnETYdQsWUQ}{Ull3Yz5HTOWwGeCOnOBApA}{172.25.0.3}{172.25.0.3:9300}{ml.machine_memory=6115733504, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])

nages · August 21, 2020, 8:43pm

Ok, I found out the issue. It seems both the containers are somehow not aware of each other and trying to create the clusters on their own.
I introduced delay with "depends", it works fine now. Thanks

system · September 18, 2020, 8:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.