ES is getting killed :(

OS:-ubuntu
ram:-16gb
Storage:-500GB
CPU:-i5 10th gen
i have Cloned ES in two different directory :
Es1
Es2

Then first i launched Es1/bin/elastisearch
And next i launched Es2/bin/elastisearch

while Es2 is launching it immediately killed the es1 process
again i try to run Es1 , Now it killed Es2 process

I even changed my port numbers

Please help me out with this :frowning:

1 Like

Please don't post images of text as they are hard to read, may not display correctly for everyone, and are not searchable.

Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

I saw that you removed the images but you are not providing the logs.

Could you provide the logs please?

kamal@PPH-LAP-00951:~/Downloads/es1$ bin/elasticsearch
[2021-07-22T15:13:47,606][INFO ][o.e.n.Node               ] [node-1] version[7.13.4], pid[5931], build[default/tar/c5f60e894ca0c61cdbae4f5a686d9f08bcefc942/2021-07-14T18:33:36.673943207Z], OS[Linux/5.8.0-59-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2021-07-22T15:13:47,609][INFO ][o.e.n.Node               ] [node-1] JVM home [/home/kamal/Downloads/es1/jdk], using bundled JDK [true]
[2021-07-22T15:13:47,610][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-13828061601626608542, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms7887m, -Xmx7887m, -XX:MaxDirectMemorySize=4135583744, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/home/kamal/Downloads/es1, -Des.path.conf=/home/kamal/Downloads/es1/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-07-22T15:13:50,355][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [aggs-matrix-stats]
[2021-07-22T15:13:50,355][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [analysis-common]
[2021-07-22T15:13:50,355][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [constant-keyword]
[2021-07-22T15:13:50,356][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [frozen-indices]
[2021-07-22T15:13:50,356][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-common]
[2021-07-22T15:13:50,356][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-geoip]
[2021-07-22T15:13:50,356][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [ingest-user-agent]
[2021-07-22T15:13:50,357][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [kibana]
[2021-07-22T15:13:50,357][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-expression]
[2021-07-22T15:13:50,357][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-mustache]
[2021-07-22T15:13:50,357][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [lang-painless]
[2021-07-22T15:13:50,358][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-extras]
[2021-07-22T15:13:50,358][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [mapper-version]
[2021-07-22T15:13:50,358][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [parent-join]
[2021-07-22T15:13:50,358][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [percolator]
[2021-07-22T15:13:50,359][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [rank-eval]
[2021-07-22T15:13:50,359][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [reindex]
[2021-07-22T15:13:50,359][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repositories-metering-api]
[2021-07-22T15:13:50,359][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-encrypted]
[2021-07-22T15:13:50,360][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [repository-url]
[2021-07-22T15:13:50,360][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [runtime-fields-common]
[2021-07-22T15:13:50,360][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [search-business-rules]
[2021-07-22T15:13:50,360][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [searchable-snapshots]
[2021-07-22T15:13:50,360][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [snapshot-repo-test-kit]
[2021-07-22T15:13:50,361][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [spatial]
[2021-07-22T15:13:50,361][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transform]
[2021-07-22T15:13:50,361][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [transport-netty4]
[2021-07-22T15:13:50,361][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [unsigned-long]
[2021-07-22T15:13:50,361][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [vectors]
[2021-07-22T15:13:50,362][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [wildcard]
[2021-07-22T15:13:50,362][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-aggregate-metric]
[2021-07-22T15:13:50,362][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-analytics]
[2021-07-22T15:13:50,362][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-async-search]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-autoscaling]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ccr]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-core]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-data-streams]
[2021-07-22T15:13:50,363][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-deprecation]
[2021-07-22T15:13:50,364][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-enrich]
[2021-07-22T15:13:50,364][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-eql]
[2021-07-22T15:13:50,364][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-fleet]
[2021-07-22T15:13:50,364][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-graph]
[2021-07-22T15:13:50,364][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-identity-provider]
[2021-07-22T15:13:50,365][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ilm]
[2021-07-22T15:13:50,365][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-logstash]
[2021-07-22T15:13:50,365][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ml]
[2021-07-22T15:13:50,365][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-monitoring]
[2021-07-22T15:13:50,366][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-ql]
[2021-07-22T15:13:50,366][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-rollup]
[2021-07-22T15:13:50,366][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-security]
[2021-07-22T15:13:50,366][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-shutdown]
[2021-07-22T15:13:50,366][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-sql]
[2021-07-22T15:13:50,367][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-stack]
[2021-07-22T15:13:50,367][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-text-structure]
[2021-07-22T15:13:50,367][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-voting-only-node]
[2021-07-22T15:13:50,367][INFO ][o.e.p.PluginsService     ] [node-1] loaded module [x-pack-watcher]
[2021-07-22T15:13:50,368][INFO ][o.e.p.PluginsService     ] [node-1] no plugins loaded
[2021-07-22T15:13:50,403][INFO ][o.e.e.NodeEnvironment    ] [node-1] using [1] data paths, mounts [[/ (/dev/mapper/vgubuntu-root)]], net usable_space [423gb], net total_space [466.2gb], types [ext4]
[2021-07-22T15:13:50,403][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [7.7gb], compressed ordinary object pointers [true]
[2021-07-22T15:13:50,487][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [ESqFWTOjRueRkDZ-eXTfZA], cluster name [my-application], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-07-22T15:13:54,576][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/6129] [Main.cc@117] controller (64 bit): Version 7.13.4 (Build 20e31b9391ed28) Copyright (c) 2021 Elasticsearch BV
[2021-07-22T15:13:54,912][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/home/kamal/Downloads/es1/config/roles.yml]
[2021-07-22T15:13:55,569][INFO ][o.e.i.g.LocalDatabases   ] [node-1] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/home/kamal/Downloads/es1/config/ingest-geoip] for changes
[2021-07-22T15:13:55,570][INFO ][o.e.i.g.DatabaseRegistry ] [node-1] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-13828061601626608542/geoip-databases/ESqFWTOjRueRkDZ-eXTfZA]
[2021-07-22T15:13:56,215][INFO ][o.e.t.NettyAllocator     ] [node-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-07-22T15:13:56,283][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2021-07-22T15:13:56,669][INFO ][o.e.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-07-22T15:13:57,065][INFO ][o.e.n.Node               ] [node-1] initialized
[2021-07-22T15:13:57,066][INFO ][o.e.n.Node               ] [node-1] starting ...
[2021-07-22T15:13:57,080][INFO ][o.e.x.s.c.f.PersistentCache] [node-1] persistent cache index loaded
[2021-07-22T15:13:57,186][INFO ][o.e.t.TransportService   ] [node-1] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-07-22T15:13:57,555][WARN ][o.e.b.BootstrapChecks    ] [node-1] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-07-22T15:13:57,556][WARN ][o.e.b.BootstrapChecks    ] [node-1] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-07-22T15:13:57,557][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [SNl-cIZ1TJSiwAZkcucS9w]
[2021-07-22T15:13:57,565][INFO ][o.e.c.c.ClusterBootstrapService] [node-1] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-07-22T15:13:57,691][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{ESqFWTOjRueRkDZ-eXTfZA}{H-fmzgzZSY-Xi_iAzy4-dw}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 18, version: 225, delta: master node changed {previous [], current [{node-1}{ESqFWTOjRueRkDZ-eXTfZA}{H-fmzgzZSY-Xi_iAzy4-dw}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2021-07-22T15:13:57,766][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{ESqFWTOjRueRkDZ-eXTfZA}{H-fmzgzZSY-Xi_iAzy4-dw}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 18, version: 225, reason: Publication{term=18, version=225}
[2021-07-22T15:13:57,820][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2021-07-22T15:13:57,821][INFO ][o.e.n.Node               ] [node-1] started
[2021-07-22T15:13:58,063][INFO ][o.e.l.LicenseService     ] [node-1] license [ba2655d1-0262-4005-8430-032b03703474] mode [basic] - valid
[2021-07-22T15:13:58,064][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled
[2021-07-22T15:13:58,065][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.13/security-minimal-setup.html to enable security.
[2021-07-22T15:13:58,068][INFO ][o.e.g.GatewayService     ] [node-1] recovered [8] indices into cluster_state
[2021-07-22T15:13:58,635][INFO ][o.e.c.r.a.AllocationService] [node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[pages][0], [.ds-ilm-history-5-2021.07.20-000001][0], [.kibana-event-log-7.13.4-000001][0]]]).
Killed
kamal@PPH-LAP-00951:~/Downloads/es1$ 

this the ES1log when i ran the ES2

kamal@PPH-LAP-00951:~/Downloads/es2$ bin/elasticsearch
[2021-07-22T15:18:37,693][INFO ][o.e.n.Node               ] [node-2] version[7.13.4], pid[6592], build[default/tar/c5f60e894ca0c61cdbae4f5a686d9f08bcefc942/2021-07-14T18:33:36.673943207Z], OS[Linux/5.8.0-59-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/16/16+36]
[2021-07-22T15:18:37,697][INFO ][o.e.n.Node               ] [node-2] JVM home [/home/kamal/Downloads/es2/jdk], using bundled JDK [true]
[2021-07-22T15:18:37,697][INFO ][o.e.n.Node               ] [node-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-14733942259271307450, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms7887m, -Xmx7887m, -XX:MaxDirectMemorySize=4135583744, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/home/kamal/Downloads/es2, -Des.path.conf=/home/kamal/Downloads/es2/config, -Des.distribution.flavor=default, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2021-07-22T15:18:40,607][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [aggs-matrix-stats]
[2021-07-22T15:18:40,608][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [analysis-common]
[2021-07-22T15:18:40,608][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [constant-keyword]
[2021-07-22T15:18:40,608][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [frozen-indices]
[2021-07-22T15:18:40,608][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-common]
[2021-07-22T15:18:40,608][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-geoip]
[2021-07-22T15:18:40,609][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [ingest-user-agent]
[2021-07-22T15:18:40,609][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [kibana]
[2021-07-22T15:18:40,609][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-expression]
[2021-07-22T15:18:40,609][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-mustache]
[2021-07-22T15:18:40,610][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [lang-painless]
[2021-07-22T15:18:40,610][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [mapper-extras]
[2021-07-22T15:18:40,610][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [mapper-version]
[2021-07-22T15:18:40,611][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [parent-join]
[2021-07-22T15:18:40,611][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [percolator]
[2021-07-22T15:18:40,611][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [rank-eval]
[2021-07-22T15:18:40,611][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [reindex]
[2021-07-22T15:18:40,612][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [repositories-metering-api]
[2021-07-22T15:18:40,612][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [repository-encrypted]
[2021-07-22T15:18:40,612][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [repository-url]
[2021-07-22T15:18:40,612][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [runtime-fields-common]
[2021-07-22T15:18:40,613][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [search-business-rules]
[2021-07-22T15:18:40,613][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [searchable-snapshots]
[2021-07-22T15:18:40,613][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [snapshot-repo-test-kit]
[2021-07-22T15:18:40,613][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [spatial]
[2021-07-22T15:18:40,614][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [transform]
[2021-07-22T15:18:40,614][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [transport-netty4]
[2021-07-22T15:18:40,614][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [unsigned-long]
[2021-07-22T15:18:40,614][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [vectors]
[2021-07-22T15:18:40,614][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [wildcard]
[2021-07-22T15:18:40,615][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-aggregate-metric]
[2021-07-22T15:18:40,615][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-analytics]
[2021-07-22T15:18:40,615][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-async]
[2021-07-22T15:18:40,615][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-async-search]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-autoscaling]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ccr]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-core]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-data-streams]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-deprecation]
[2021-07-22T15:18:40,616][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-enrich]
[2021-07-22T15:18:40,617][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-eql]
[2021-07-22T15:18:40,617][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-fleet]
[2021-07-22T15:18:40,617][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-graph]
[2021-07-22T15:18:40,617][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-identity-provider]
[2021-07-22T15:18:40,618][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ilm]
[2021-07-22T15:18:40,618][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-logstash]
[2021-07-22T15:18:40,618][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ml]
[2021-07-22T15:18:40,618][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-monitoring]
[2021-07-22T15:18:40,619][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-ql]
[2021-07-22T15:18:40,619][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-rollup]
[2021-07-22T15:18:40,619][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-security]
[2021-07-22T15:18:40,619][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-shutdown]
[2021-07-22T15:18:40,619][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-sql]
[2021-07-22T15:18:40,620][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-stack]
[2021-07-22T15:18:40,620][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-text-structure]
[2021-07-22T15:18:40,620][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-voting-only-node]
[2021-07-22T15:18:40,620][INFO ][o.e.p.PluginsService     ] [node-2] loaded module [x-pack-watcher]
[2021-07-22T15:18:40,621][INFO ][o.e.p.PluginsService     ] [node-2] no plugins loaded
[2021-07-22T15:18:40,660][INFO ][o.e.e.NodeEnvironment    ] [node-2] using [1] data paths, mounts [[/ (/dev/mapper/vgubuntu-root)]], net usable_space [423gb], net total_space [466.2gb], types [ext4]
[2021-07-22T15:18:40,660][INFO ][o.e.e.NodeEnvironment    ] [node-2] heap size [7.7gb], compressed ordinary object pointers [true]
[2021-07-22T15:18:40,746][INFO ][o.e.n.Node               ] [node-2] node name [node-2], node ID [ESqFWTOjRueRkDZ-eXTfZA], cluster name [my-application], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-07-22T15:18:45,163][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-2] [controller/6795] [Main.cc@117] controller (64 bit): Version 7.13.4 (Build 20e31b9391ed28) Copyright (c) 2021 Elasticsearch BV
[2021-07-22T15:18:45,608][INFO ][o.e.x.s.a.s.FileRolesStore] [node-2] parsed [0] roles from file [/home/kamal/Downloads/es2/config/roles.yml]
[2021-07-22T15:18:46,316][INFO ][o.e.i.g.LocalDatabases   ] [node-2] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/home/kamal/Downloads/es2/config/ingest-geoip] for changes
[2021-07-22T15:18:46,317][INFO ][o.e.i.g.DatabaseRegistry ] [node-2] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-14733942259271307450/geoip-databases/ESqFWTOjRueRkDZ-eXTfZA]
[2021-07-22T15:18:47,004][INFO ][o.e.t.NettyAllocator     ] [node-2] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-07-22T15:18:47,067][INFO ][o.e.d.DiscoveryModule    ] [node-2] using discovery type [zen] and seed hosts providers [settings]
[2021-07-22T15:18:47,525][INFO ][o.e.g.DanglingIndicesState] [node-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-07-22T15:18:47,920][INFO ][o.e.n.Node               ] [node-2] initialized
[2021-07-22T15:18:47,920][INFO ][o.e.n.Node               ] [node-2] starting ...
[2021-07-22T15:18:47,933][INFO ][o.e.x.s.c.f.PersistentCache] [node-2] persistent cache index loaded
[2021-07-22T15:18:48,034][INFO ][o.e.t.TransportService   ] [node-2] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2021-07-22T15:18:48,413][WARN ][o.e.b.BootstrapChecks    ] [node-2] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-07-22T15:18:48,413][WARN ][o.e.b.BootstrapChecks    ] [node-2] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-07-22T15:18:48,414][INFO ][o.e.c.c.Coordinator      ] [node-2] cluster UUID [SNl-cIZ1TJSiwAZkcucS9w]
[2021-07-22T15:18:48,424][INFO ][o.e.c.c.ClusterBootstrapService] [node-2] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-07-22T15:18:48,542][INFO ][o.e.c.s.MasterService    ] [node-2] elected-as-master ([1] nodes joined)[{node-2}{ESqFWTOjRueRkDZ-eXTfZA}{dUtsGodcT0-vRzEs1TvbAA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 17, version: 216, delta: master node changed {previous [], current [{node-2}{ESqFWTOjRueRkDZ-eXTfZA}{dUtsGodcT0-vRzEs1TvbAA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2021-07-22T15:18:48,616][INFO ][o.e.c.s.ClusterApplierService] [node-2] master node changed {previous [], current [{node-2}{ESqFWTOjRueRkDZ-eXTfZA}{dUtsGodcT0-vRzEs1TvbAA}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 17, version: 216, reason: Publication{term=17, version=216}
[2021-07-22T15:18:48,667][INFO ][o.e.h.AbstractHttpServerTransport] [node-2] publish_address {127.0.0.1:9201}, bound_addresses {[::1]:9201}, {127.0.0.1:9201}
[2021-07-22T15:18:48,667][INFO ][o.e.n.Node               ] [node-2] started
[2021-07-22T15:18:48,963][INFO ][o.e.l.LicenseService     ] [node-2] license [ba2655d1-0262-4005-8430-032b03703474] mode [basic] - valid
[2021-07-22T15:18:48,965][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Active license is now [BASIC]; Security is disabled
[2021-07-22T15:18:48,965][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [node-2] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.13/security-minimal-setup.html to enable security.
[2021-07-22T15:18:48,970][INFO ][o.e.g.GatewayService     ] [node-2] recovered [8] indices into cluster_state
[2021-07-22T15:18:49,533][INFO ][o.e.c.r.a.AllocationService] [node-2] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana-event-log-7.13.4-000001][0], [.ds-ilm-history-5-2021.07.20-000001][0]]]).




these are the ES2 logs

so When i run ES2 its killing the process of ES1
i have changed the port too

What does the config files look like?

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9201
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

this is ES2

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

this is ES1

please anyone guide me through fixing this

What's most likely happening here is that the second node is being killed by the system OOM killer. By default Elasticsearch automatically determines appropriate JVM heap size based on total system memory. Part of this heuristic is the assumption that only one Elasticsearch node will be running on the system at a time. We can see from the reported JVM argument -Xms7887m that Elasticsearch is using nearly 8GB of heap so with two nodes running we quickly exhaust the total system memory of 16GB. I'd suggest you manually configure each node's heap to ensure you stay within system limits.

5 Likes

thanks alot it helped me . :smiley:

Omg so that explains a lot!

I was trying to run a demo with multiple nodes locally, using the data.path option as always:

# Terminal 1
./bin/elasticsearch

# Terminal 2
./bin/elasticsearch -Epath.data=/tmp/esbis

This was working fine on other 7.x versions (at least on 7.5). But on 7.13, running the second node was always killing the first one, with no log / no explanation.

The reason being that the default HEAP setting has been changed from 1 Gb to half the available memory by default (Autodetermine heap settings based on node roles and total system memory by mark-vieira · Pull Request #65905 · elastic/elasticsearch · GitHub).

The solution is to force a lower HEAP:

ES_JAVA_OPTS="-Xms1g -Xmx1g" ./bin/elasticsearch

Thanks for the solution - still not sure why my main node is getting killed when trying to start a second one tho, I would have expected the second one to just refuse starting by lack of available HEAP, not kill and replace the existing node ^^

1 Like