Elasticsearch failed to start on Ubuntu 18.04

Hi folks,

I am an elastic noob and I am stuck with running elasticsearch on an Ubuntu 18.04 V-Server with 4 Cores, 8Gb Ram and a 100GB Hard drive. I followed the instructions from this post: Installieren und Konfigurieren von Elasticsearch unter Ubuntu 18.04 | DigitalOcean and I end up with the following error:

The errors says OutOfMemory, but it should be fine with 8GB?
If anyone could help me, would be awesome..

Kind regards!

Welcome to our community! :smiley:

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

Ah sorry my bad.
So after using the command systemctl status elasticsearch.service I get the following error log:

â—Ź elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-03-25 05:41:06 CET; 3s ago
Docs: https://www.elastic.co
Process: 12434 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code
Main PID: 12434 (code=exited, status=127)
Tasks: 0 (limit: 60)
CGroup: /system.slice/elasticsearch.service

Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: java.lang.OutOfMemoryError: unable to create native
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.start0(Native
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.start(Thread.
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.util.concurrent.ThreadPoo
Mär 25 05:41:05 h2920142.stratoserver.net systemd-entrypoint[12434]: at java.base/java.lang.Thread.run(Thread.ja
Mär 25 05:41:06 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=12
Mär 25 05:41:06 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Kind regards :slight_smile:

Did you change the memory options? If so, check that you don't use more than 4gb of HEAP and that this memory is available on your machine (meaning that no other service is running).

Hi David, I kept the elastic default config. My machine has about 7.9GB RAM free

What is the free memory available and which version are you running ?

I have about 94Gb free disk spaces. My elastic version is 7.12.0:
root@h2920142:/usr/share/elasticsearch# bin/elasticsearch --version
Version: 7.12.0, Build: default/deb/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z, JVM: 15.0.1

I'm asking about the free RAM memory not the free disk size.

ah ok, ye I told that in the previous post. Its 7.9GB

using free -m

gesamt belegt frei gemeinsam Zwischen verfĂĽgbar
Speicher: 8192 108 7918 3 164 8079
Auslager: 0 0 0

Could you try change the JVM memory settings?
In 7.12 elasticsearch tries to guess this value for you but may be there's something wrong here.

You should rarely need to change Java Virtual Machine (JVM) options. Elasticsearch includes default JVM options that work well for most production environments. If needed, you can override these default options using jvm.options files or the ES_JAVA_OPTS environment variable.

See Setting JVM options | Elasticsearch Guide [8.11] | Elastic and Advanced configuration | Elasticsearch Guide [8.11] | Elastic

Try with something like -Xmx1g -Xms1g to start with.

Also do you see any elasticsearch logs?

I have created a jvm.options file under the jvm.options.d folder and added the line -Xmx1g -Xms1g. Is this correct?
I have the same error still.
My logs look like this: tail -f /var/log/elasticsearch/elasticsearch.log:

elasticsearch.service: Main process exited, code=exited, status=12
Mär 25 10:22:16 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mär 25 10:22:16 h2920142.stratoserver.net systemd[1]: Failed to start Elasticsearch.
root@h2920142:/etc/elasticsearch/jvm.options.d#
root@h2920142:/etc/elasticsearch/jvm.options.d# sudo systemctl start elasticsearch
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
root@h2920142:/etc/elasticsearch/jvm.options.d# tail -f /var/log/elasticsearch/elasticsearch.log
[2021-03-25T10:25:07,548][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T10:25:07,631][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][scheduler][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:801) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

When i enter journalctl -xe: I get

Mär 25 10:33:38 h2920142.stratoserver.net sshd[16833]: Received disconnect from 161.35.10.226 port 40924:11: Bye Bye [preauth]
Mär 25 10:33:38 h2920142.stratoserver.net sshd[16833]: Disconnected from authenticating user root 161.35.10.226 port 40924 [preauth]
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: [18,989s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 102
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: [18,990s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 102
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reache
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.start0(Native Method)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.start(Thread.java:801)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
Mär 25 10:33:42 h2920142.stratoserver.net systemd-entrypoint[16633]:         at java.base/java.lang.Thread.run(Thread.java:832)
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mär 25 10:33:43 h2920142.stratoserver.net systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit elasticsearch.service has failed.
--
-- The result is RESULT.

Could use the markdown format for code and logs? Instead of <code>?
I edited your post so you can see how this should be done.

Could you share the full /var/log/elasticsearch/elasticsearch.log?

ah sorry, yes I ll try the markdown :slight_smile:

[2021-03-25T10:33:30,990][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-watcher]
[2021-03-25T10:33:30,991][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] no plugins loaded
[2021-03-25T10:33:31,039][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93.4gb], net total_space [100.8gb], types [rootfs]
[2021-03-25T10:33:31,043][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] heap size [4gb], compressed ordinary object pointers [true]
[2021-03-25T10:33:31,139][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] node name [h2920142.stratoserver.net], node ID [tuImGnRMSdqZi3yT-5iUAg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-03-25T10:33:37,531][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [h2920142.stratoserver.net] [controller/16826] [Main.cc@117] controller (64 bit): Version 7.12.0 (Build 778cc7261dfdd1) Copyright (c) 2021 Elasticsearch BV
[2021-03-25T10:33:38,247][INFO ][o.e.x.s.a.s.FileRolesStore] [h2920142.stratoserver.net] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-03-25T10:33:40,231][INFO ][o.e.t.NettyAllocator     ] [h2920142.stratoserver.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-03-25T10:33:40,318][INFO ][o.e.d.DiscoveryModule    ] [h2920142.stratoserver.net] using discovery type [zen] and seed hosts providers [settings]
[2021-03-25T10:33:40,944][INFO ][o.e.g.DanglingIndicesState] [h2920142.stratoserver.net] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-03-25T10:33:41,572][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] initialized
[2021-03-25T10:33:41,572][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] starting ...
[2021-03-25T10:33:41,594][INFO ][o.e.x.s.c.PersistentCache] [h2920142.stratoserver.net] persistent cache index loaded
[2021-03-25T10:33:41,666][WARN ][i.n.u.i.MacAddressUtil   ] [h2920142.stratoserver.net] Failed to find a usable hardware address from the network interfaces; using random bytes: 91:ec:3b:93:c9:90:de:2d
[2021-03-25T10:33:41,717][INFO ][o.e.t.TransportService   ] [h2920142.stratoserver.net] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2021-03-25T10:33:41,860][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-03-25T10:33:41,860][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-03-25T10:33:41,862][INFO ][o.e.c.c.Coordinator      ] [h2920142.stratoserver.net] cluster UUID [2vu1TGTuTCedtYgd_KJXLg]
[2021-03-25T10:33:41,880][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T10:33:42,064][INFO ][o.e.c.s.MasterService    ] [h2920142.stratoserver.net] elected-as-master ([1] nodes joined)[{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 16, version: 19, delta: master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}
[2021-03-25T10:33:42,122][INFO ][o.e.c.s.ClusterApplierService] [h2920142.stratoserver.net] master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{F-E96mN_TcaeSXivK_tvyQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}, term: 16, version: 19, reason: Publication{term=16, version=19}
[2021-03-25T10:33:42,178][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
        at java.lang.Thread.start0(Native Method) ~[?:?]
        at java.lang.Thread.start(Thread.java:801) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

Could you share the full logs please?

I am trying, but its not allowed due to the max amount of chars per post: Here is the first part of the full logs:

[2021-03-25T12:47:08,111][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] version[7.12.0], pid[18518], build[default/deb/78722783c38caa25a70982b5b042074cde5d3b3a/2021-03-18T06:17:15.410153305Z], OS[Linux/4.15.0/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2021-03-25T12:47:08,126][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2021-03-25T12:47:08,126][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-3286324001059924212, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms4096m, -Xmx4096m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb, -Des.bundled_jdk=true]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [aggs-matrix-stats]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [analysis-common]
[2021-03-25T12:47:12,940][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [constant-keyword]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [flattened]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [frozen-indices]
[2021-03-25T12:47:12,941][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-common]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-geoip]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [ingest-user-agent]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [kibana]
[2021-03-25T12:47:12,942][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-expression]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-mustache]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [lang-painless]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [mapper-extras]
[2021-03-25T12:47:12,943][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [mapper-version]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [parent-join]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [percolator]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [rank-eval]
[2021-03-25T12:47:12,944][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [reindex]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repositories-metering-api]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repository-encrypted]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [repository-url]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [search-business-rules]
[2021-03-25T12:47:12,945][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [searchable-snapshots]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [snapshot-repo-test-kit]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [spatial]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [systemd]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [transform]
[2021-03-25T12:47:12,950][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [transport-netty4]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [unsigned-long]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [vectors]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [wildcard]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-aggregate-metric]
[2021-03-25T12:47:12,951][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-analytics]
[2021-03-25T12:47:12,952][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-async]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-async-search]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-autoscaling]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ccr]
[2021-03-25T12:47:12,954][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-core]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-data-streams]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-deprecation]
[2021-03-25T12:47:12,955][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-enrich]
[2021-03-25T12:47:12,956][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-eql]
[2021-03-25T12:47:12,956][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-fleet]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-graph]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-identity-provider]
[2021-03-25T12:47:12,958][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ilm]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ingest]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-logstash]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ml]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-monitoring]
[2021-03-25T12:47:12,959][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-ql]
[2021-03-25T12:47:12,960][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-rollup]
[2021-03-25T12:47:12,960][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-runtime-fields]
[2021-03-25T12:47:12,962][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-security]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-sql]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-stack]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-text-structure]
[2021-03-25T12:47:12,963][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-voting-only-node]
[2021-03-25T12:47:12,964][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] loaded module [x-pack-watcher]
[2021-03-25T12:47:12,964][INFO ][o.e.p.PluginsService     ] [h2920142.stratoserver.net] no plugins loaded

Here is the second part

[2021-03-25T12:47:13,032][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [93.4gb], net total_space [100.8gb], types [rootfs]
[2021-03-25T12:47:13,032][INFO ][o.e.e.NodeEnvironment    ] [h2920142.stratoserver.net] heap size [4gb], compressed ordinary object pointers [true]
[2021-03-25T12:47:13,160][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] node name [h2920142.stratoserver.net], node ID [tuImGnRMSdqZi3yT-5iUAg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-03-25T12:47:20,384][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [h2920142.stratoserver.net] [controller/18712] [Main.cc@117] controller (64 bit): Version 7.12.0 (Build 778cc7261dfdd1) Copyright (c) 2021 Elasticsearch BV
[2021-03-25T12:47:21,289][INFO ][o.e.x.s.a.s.FileRolesStore] [h2920142.stratoserver.net] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-03-25T12:47:24,232][INFO ][o.e.t.NettyAllocator     ] [h2920142.stratoserver.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-03-25T12:47:24,320][INFO ][o.e.d.DiscoveryModule    ] [h2920142.stratoserver.net] using discovery type [zen] and seed hosts providers [settings]
[2021-03-25T12:47:25,309][INFO ][o.e.g.DanglingIndicesState] [h2920142.stratoserver.net] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-03-25T12:47:25,999][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] initialized
[2021-03-25T12:47:25,999][INFO ][o.e.n.Node               ] [h2920142.stratoserver.net] starting ...
[2021-03-25T12:47:26,009][INFO ][o.e.x.s.c.PersistentCache] [h2920142.stratoserver.net] persistent cache index loaded
[2021-03-25T12:47:26,085][WARN ][i.n.u.i.MacAddressUtil   ] [h2920142.stratoserver.net] Failed to find a usable hardware address from the network interfaces; using random bytes: 16:c0:4c:0a:de:73:22:2a
[2021-03-25T12:47:26,136][INFO ][o.e.t.TransportService   ] [h2920142.stratoserver.net] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2021-03-25T12:47:26,278][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2021-03-25T12:47:26,279][WARN ][o.e.b.BootstrapChecks    ] [h2920142.stratoserver.net] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-03-25T12:47:26,280][INFO ][o.e.c.c.Coordinator      ] [h2920142.stratoserver.net] cluster UUID [2vu1TGTuTCedtYgd_KJXLg]
[2021-03-25T12:47:26,303][INFO ][o.e.c.c.ClusterBootstrapService] [h2920142.stratoserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-03-25T12:47:26,489][INFO ][o.e.c.s.MasterService    ] [h2920142.stratoserver.net] elected-as-master ([1] nodes joined)[{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 17, version: 20, delta: master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}
[2021-03-25T12:47:26,540][INFO ][o.e.c.s.ClusterApplierService] [h2920142.stratoserver.net] master node changed {previous [], current [{h2920142.stratoserver.net}{tuImGnRMSdqZi3yT-5iUAg}{h3FyRTqQS36nplxca8QWYQ}{localhost}{127.0.0.1:9300}{cdfhilmrstw}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=4294967296}]}, term: 17, version: 20, reason: Publication{term=17, version=20}
[2021-03-25T12:47:26,591][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [h2920142.stratoserver.net] fatal error in thread [elasticsearch[h2920142.stratoserver.net][clusterApplierService#updateTask][T#1]], exiting
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.lang.Thread.start0(Native Method) ~[?:?]
	at java.lang.Thread.start(Thread.java:801) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:939) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1007) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

This could be your problem
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Try increasing it and that should help.

1 Like

If you don't use it, I'd disable machine learning.
But I feel like there's a bug. I'm checking a bit more.

How much did you set to the heap? Apparently it's using 4gb.

I tried to raise the memory but it had no impact on the error