Elasticsearch failed to start on Ubuntu 18.04

Sebastian_F · March 24, 2021, 8:00pm

Hi folks,

I am an elastic noob and I am stuck with running elasticsearch on an Ubuntu 18.04 V-Server with 4 Cores, 8Gb Ram and a 100GB Hard drive. I followed the instructions from this post: Installieren und Konfigurieren von Elasticsearch unter Ubuntu 18.04 | DigitalOcean and I end up with the following error:

The errors says OutOfMemory, but it should be fine with 8GB?
If anyone could help me, would be awesome..

Kind regards!

warkolm · March 24, 2021, 8:18pm

Welcome to our community!

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

Sebastian_F · March 25, 2021, 4:45am

Ah sorry my bad.
So after using the command systemctl status elasticsearch.service I get the following error log:

● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2021-03-25 05:41:06 CET; 3s ago Docs: https://www.elastic.co Process: 12434 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code Main PID: 12434 (code=exited, status=127) Tasks: 0 (limit: 60) CGroup: /system.slice/elasticsearch.service

Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: java.lang.OutOfMemoryError: unable to create native Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.start0(Native Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.start(Thread. Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.run(Thread.ja Mär 25 05:41:06 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=12 Mär 25 05:41:06 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Kind regards

dadoonet · March 25, 2021, 6:10am

Did you change the memory options? If so, check that you don't use more than 4gb of HEAP and that this memory is available on your machine (meaning that no other service is running).

Sebastian_F · March 25, 2021, 6:33am

Hi David, I kept the elastic default config. My machine has about 7.9GB RAM free

dadoonet · March 25, 2021, 6:44am

What is the free memory available and which version are you running ?

Sebastian_F · March 25, 2021, 7:46am

I have about 94Gb free disk spaces. My elastic version is 7.12.0:
root@h2920142:/usr/share/elasticsearch# bin/elasticsearch --version Version: 7.12.0, Build: default/deb/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z, JVM: 15.0.1

dadoonet · March 25, 2021, 7:52am

I'm asking about the free RAM memory not the free disk size.

Sebastian_F · March 25, 2021, 7:54am

ah ok, ye I told that in the previous post. Its 7.9GB

Sebastian_F · March 25, 2021, 7:56am

using free -m

gesamt belegt frei gemeinsam Zwischen verfügbar Speicher: 8192 108 7918 3 164 8079 Auslager: 0 0 0

dadoonet · March 25, 2021, 8:50am

Could you try change the JVM memory settings?
In 7.12 elasticsearch tries to guess this value for you but may be there's something wrong here.

You should rarely need to change Java Virtual Machine (JVM) options. Elasticsearch includes default JVM options that work well for most production environments. If needed, you can override these default options using jvm.options files or the ES_JAVA_OPTS environment variable.

See Setting JVM options | Elasticsearch Guide [8.11] | Elastic and Advanced configuration | Elasticsearch Guide [8.11] | Elastic

Try with something like -Xmx1g -Xms1g to start with.

Also do you see any elasticsearch logs?

Sebastian_F · March 25, 2021, 9:35am

I have created a jvm.options file under the jvm.options.d folder and added the line -Xmx1g -Xms1g. Is this correct?
I have the same error still.
My logs look like this: tail -f /var/log/elasticsearch/elasticsearch.log:

elasticsearch.service: Main process exited, code=exited, status=12
Mär 25 10:22:16 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mär 25 10:22:16 h2920142.stratoserver.net systemd[1]: Failed to start Elasticsearch.
root@h2920142:/etc/elasticsearch/jvm.options.d#
root@h2920142:/etc/elasticsearch/jvm.options.d# sudo systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
root@h2920142:/etc/elasticsearch/jvm.options.d# tail -f /var/log/elasticsearch/elasticsearch.log
[2021-03-25T10:25:07,548][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T10:25:07,631][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][scheduler][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:801) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

When i enter journalctl -xe: I get

Mär 25 10:33:38 h2920142.stratoserver.net sshd[16833]: Received disconnect from 161.35.10.226 port 40924:11: Bye Bye [preauth]
Mär 25 10:33:38 h2920142.stratoserver.net sshd[16833]: Disconnected from authenticating user root 161.35.10.226 port 40924 [preauth]
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: [18,989s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 102
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: [18,990s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 102
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reache
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.start0(Native Method)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.start(Thread.java:801)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.run(Thread.java:832)
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit elasticsearch.service has failed.
--
-- The result is RESULT.

dadoonet · March 25, 2021, 10:02am

Could use the markdown format for code and logs? Instead of <code>?
I edited your post so you can see how this should be done.

Could you share the full /var/log/elasticsearch/elasticsearch.log?

Sebastian_F · March 25, 2021, 10:32am

ah sorry, yes I ll try the markdown

[2021-03-25T10:33:30,990][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-watcher]
[2021-03-25T10:33:30,991][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] no plugins loaded
[2021-03-25T10:33:31,039][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93.4gb], net total_space [100.8gb], types [rootfs]
[2021-03-25T10:33:31,043][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] heap size [4gb], compressed ordinary object pointers [true]
[2021-03-25T10:33:31,139][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] node name [h2920142.stratoserver.net], node ID [tuImGnRMSdqZi3yT-5iUAg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-03-25T10:33:37,531][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [h2920142.stratoserver.net] [controller/16826] [Main.cc@117] controller (64 bit): Version 7.12.0 (Build 778cc7261dfdd1) Copyright (c) 2021 Elasticsearch BV
[2021-03-25T10:33:38,247][INFO ][o.e.x.s.a.s.FileRolesStore] [h2920142.stratoserver.net] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-03-25T10:33:40,231][INFO ][o.e.t.NettyAllocator     ] [h2920142.stratoserver.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-03-25T10:33:40,318][INFO ][o.e.d.DiscoveryModule    ] [h2920142.stratoserver.net] using discovery type [zen] and seed hosts providers [settings]
[2021-03-25T10:33:40,944][INFO ][o.e.g.DanglingIndicesState] [h2920142.stratoserver.net] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-03-25T10:33:41,572][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] initialized
[2021-03-25T10:33:41,572][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] starting ...
[2021-03-25T10:33:41,594][INFO ][o.e.x.s.c.PersistentCache] [h2920142.stratoserver.net] persistent cache index loaded
[2021-03-25T10:33:41,666][WARN ][i.n.u.i.MacAddressUtil   ] [h2920142.stratoserver.net] Failed to find a usable hardware address from the network interfaces; using random bytes: 91:ec:3b:93:c9:90:de:2d
[2021-03-25T10:33:41,717][INFO ][o.e.t.TransportService   ] [h2920142.stratoserver.net] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2021-03-25T10:33:41,860][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-03-25T10:33:41,860][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-03-25T10:33:41,862][INFO ][o.e.c.c.Coordinator      ] [h2920142.stratoserver.net] cluster UUID [2vu1TGTuTCedtYgd_KJXLg]
[2021-03-25T10:33:41,880][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T10:33:42,064][INFO ][o.e.c.s.MasterService    ] [h2920142.stratoserver.net] elected-as-master ([1] nodes joined)[{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 16, version: 19, delta: master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}
[2021-03-25T10:33:42,122][INFO ][o.e.c.s.ClusterApplierService] [h2920142.stratoserver.net] master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}, term: 16, version: 19, reason: Publication{term=16, version=19}
[2021-03-25T10:33:42,178][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:801) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

dadoonet · March 25, 2021, 11:26am

Could you share the full logs please?

Sebastian_F · March 25, 2021, 11:53am

I am trying, but its not allowed due to the max amount of chars per post: Here is the first part of the full logs:

[2021-03-25T12:47:08,111][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] version[7.12.0], pid[18518], build[default/deb/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z], OS[Linux/4.15.0/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-03-25T12:47:08,126][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-03-25T12:47:08,126][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-3286324001059924212, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms4096m, -Xmx4096m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [aggs-matrix-stats]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [analysis-common]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [constant-keyword]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [flattened]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [frozen-indices]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-common]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-geoip]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-user-agent]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [kibana]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-expression]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-mustache]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-painless]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [mapper-extras]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [mapper-version]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [parent-join]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [percolator]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [rank-eval]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [reindex]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repositories-metering-api]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repository-encrypted]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repository-url]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [search-business-rules]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [searchable-snapshots]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [snapshot-repo-test-kit]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [spatial]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [systemd]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [transform]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [transport-netty4]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [unsigned-long]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [vectors]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [wildcard]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-aggregate-metric]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-analytics]
[2021-03-25T12:47:12,952][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-async]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-async-search]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-autoscaling]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ccr]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-core]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-data-streams]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-deprecation]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-enrich]
[2021-03-25T12:47:12,956][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-eql]
[2021-03-25T12:47:12,956][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-fleet]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-graph]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-identity-provider]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ilm]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ingest]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-logstash]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ml]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-monitoring]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ql]
[2021-03-25T12:47:12,960][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-rollup]
[2021-03-25T12:47:12,960][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-runtime-fields]
[2021-03-25T12:47:12,962][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-security]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-sql]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-stack]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-text-structure]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-voting-only-node]
[2021-03-25T12:47:12,964][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-watcher]
[2021-03-25T12:47:12,964][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] no plugins loaded

Sebastian_F · March 25, 2021, 11:54am

Here is the second part

[2021-03-25T12:47:13,032][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93.4gb], net total_space [100.8gb], types [rootfs]
[2021-03-25T12:47:13,032][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] heap size [4gb], compressed ordinary object pointers [true]
[2021-03-25T12:47:13,160][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] node name [h2920142.stratoserver.net], node ID [tuImGnRMSdqZi3yT-5iUAg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-03-25T12:47:20,384][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [h2920142.stratoserver.net] [controller/18712] [Main.cc@117] controller (64 bit): Version 7.12.0 (Build 778cc7261dfdd1) Copyright (c) 2021 Elasticsearch BV
[2021-03-25T12:47:21,289][INFO ][o.e.x.s.a.s.FileRolesStore] [h2920142.stratoserver.net] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-03-25T12:47:24,232][INFO ][o.e.t.NettyAllocator     ] [h2920142.stratoserver.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-03-25T12:47:24,320][INFO ][o.e.d.DiscoveryModule    ] [h2920142.stratoserver.net] using discovery type [zen] and seed hosts providers [settings]
[2021-03-25T12:47:25,309][INFO ][o.e.g.DanglingIndicesState] [h2920142.stratoserver.net] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-03-25T12:47:25,999][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] initialized
[2021-03-25T12:47:25,999][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] starting ...
[2021-03-25T12:47:26,009][INFO ][o.e.x.s.c.PersistentCache] [h2920142.stratoserver.net] persistent cache index loaded
[2021-03-25T12:47:26,085][WARN ][i.n.u.i.MacAddressUtil   ] [h2920142.stratoserver.net] Failed to find a usable hardware address from the network interfaces; using random bytes: 16:c0:4c:0a:de:73:22:2a
[2021-03-25T12:47:26,136][INFO ][o.e.t.TransportService   ] [h2920142.stratoserver.net] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2021-03-25T12:47:26,278][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-03-25T12:47:26,279][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-03-25T12:47:26,280][INFO ][o.e.c.c.Coordinator      ] [h2920142.stratoserver.net] cluster UUID [2vu1TGTuTCedtYgd_KJXLg]
[2021-03-25T12:47:26,303][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T12:47:26,489][INFO ][o.e.c.s.MasterService    ] [h2920142.stratoserver.net] elected-as-master ([1] nodes joined)[{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 17, version: 20, delta: master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}
[2021-03-25T12:47:26,540][INFO ][o.e.c.s.ClusterApplierService] [h2920142.stratoserver.net] master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}, term: 17, version: 20, reason: Publication{term=17, version=20}
[2021-03-25T12:47:26,591][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:801) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

Sreekanth_Ragi · March 30, 2021, 6:30am

This could be your problem
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Try increasing it and that should help.

dadoonet · March 30, 2021, 6:48am

If you don't use it, I'd disable machine learning.
But I feel like there's a bug. I'm checking a bit more.

How much did you set to the heap? Apparently it's using 4gb.

Sebastian_F · March 31, 2021, 5:22pm

I tried to raise the memory but it had no impact on the error