Hi there, can somebody tell me please why my Elastic installation dont want to start? Log says it is out of memory. But it should contains only one index with about one million items. The server params are: 6vCPU, 16G RAM, 400G SSD
What should I check for?
Here is the result of journalctl -xe
Aug 27 23:49:21 vmi503579.contaboserver.net kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=526834,uid=111
Aug 27 23:49:21 vmi503579.contaboserver.net kernel: Out of memory: Killed process 526834 (java) total-vm:9572128kB, anon-rss:8405852kB, file-rss:12kB, shmem-rss:0kB, UID:111 pgtables:16684kB oom_score_adj:0
Aug 27 23:49:21 vmi503579.contaboserver.net kernel: oom_reaper: reaped process 526834 (java), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Aug 27 23:49:21 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit elasticsearch.service has exited.
--
-- The process' exit code is 'killed' and its exit status is 9.
Aug 27 23:49:21 vmi503579.contaboserver.net systemd[1]: elasticsearch.service: Failed with result 'signal'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit elasticsearch.service has entered the 'failed' state with result 'signal'.
Aug 27 23:49:21 vmi503579.contaboserver.net systemd[1]: Failed to start Elasticsearch.
-- Subject: A start job for unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit elasticsearch.service has finished with a failure.
--
-- The job identifier is 2627624 and the job result is failed.
Aug 27 23:49:44 vmi503579.contaboserver.net kernel: [UFW BLOCK] IN=eth0 OUT= MAC=00:50:56:41:58:b2:28:99:3a:4d:23:91:08:00 SRC=193.27.229.74 DST=173.249.58.30 LEN=40 TOS=0x00 PREC=0x00 TTL=247 ID=51905 PROTO=TCP SPT=55505
the server runs Apache with few webs, Mysql, PHP, Jenkins and some Selenium tests at night. Everything works fine.
Elastic version is 7
Elastic run before, but also had some problems.
Dont understand what you mean by java options. I run it via systemctl start elasticsearch.
Here is the Elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: localhost
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
Java options is the jvm.options, it is the file where you set how much memory elasticsearch should use for the Java Heap, the options -Xmx and -Xms.
How much memory are configured for elasticsearch?
You are running a lot of things in the same server, the recomendation is to run elasticsearch alone in the server and give it about half of the memory, you should also disable swapping, which could impact the other applications if run in the same server.
@leandrojmp Server is Ubuntu 20.04 6vCPU 16G Ram 400G SSD.
Xms and xmx options are commented in jvm.options
################################################################
##
## JVM configuration
##
################################################################
##
## WARNING: DO NOT EDIT THIS FILE. If you want to override the
## JVM options in this file, or set any additional options, you
## should create one or more files in the jvm.options.d
## directory containing your adjustments.
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/jvm-options.html
## for more information.
##
################################################################
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## The heap size is automatically configured by Elasticsearch
## based on the available memory in your system and the roles
## each node is configured to fulfill. If specifying heap is
## required, it should be done through a file in jvm.options.d,
## and the min and max should be set to the same value. For
## example, to set the heap to 4 GB, create a new file in the
## jvm.options.d directory containing these lines:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
################################################################
## Expert settings
################################################################
##
## All settings below here are considered expert settings. Do
## not adjust them unless you understand what you are doing. Do
## not edit them in this file; instead, create a new file in the
## jvm.options.d directory containing your adjustments.
##
################################################################
## GC configuration
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
## G1GC Configuration
# NOTE: G1 GC is only supported on JDK version 10 or later
# to use G1GC, uncomment the next two lines and update the version on the
# following three lines to your version of the JDK
# 10-13:-XX:-UseConcMarkSweepGC
# 10-13:-XX:-UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
## JVM temporary directory
-Djava.io.tmpdir=${ES_TMPDIR}
## heap dumps
# generate a heap dump when an allocation from the Java heap fails; heap dumps
# are created in the working directory of the JVM unless an alternative path is
# specified
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps; ensure the directory exists and
# has sufficient space
-XX:HeapDumpPath=/var/lib/elasticsearch
# specify an alternative path for JVM fatal error logs
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
## JDK 8 GC logging
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
# JDK 9+ GC logging
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m
What it means? It should be set up automatically by Elasticsearch while installation according available resources. Shouldn't be?
Since version 7.11 elasticsearch can automatically set the heap size, from the documentation this calculation is made based on the node's role and total memory, I don't think it will take in consideration other apps running as the recommendation is to run elasticsearch alone on the server.
Since you have a couple of other apps running this could impact your elasticsearch service, maybe it is calculating a size for the heap, but can't get the memory because other programs are already using it.
In this case I would recommend that you manually set the heap size to a fixed value, use the same value in Xmx and Xms, this value will depend on your specs that you didn't share.
I believe it sets it automatically based on the server memory and the assumption that it has sole access to all of it (recommended best practice), which is not the case as you are hosting other services on it. I therefore think you will need to set it manually.
Please what specs should I share? I dont really know. It has so many params I am lost in it. I though cpu and the ram are the main part of this problem. What am I missing? What number should be the xmx xms values?
Try setting this in the jvm.options I believe that used to be the old default that's a pretty small setting and you can start to work your way up from there
-Xms1g
-Xmx1g
Don't change any other settings in that file.
But in reality it depends on the number of indexes in shards etc.
Let's see if we can get it up and running then go from there
@stephenb it works. Also other services are not slow down. Is there any other problem which could shut it down? I dont know some Elastic internal processes?
It is not best practice nor ever recommended to run elasticsearch on a host or VM with other major processes. Doing so always creates more opportunities for issues.
With that in mind absolutely there could be other issues but I can't name them all but most of them would be related to not enough resources or sharing resources.
Your best option would be to put elasticsearch on its own host or VM
It is for educational purposes only. It would be little overkill to have separate machine for my experiments with Elastic. Hope it can run in this scenario.
Understood I am just relating best practices and why you are having issues.
Elasticsearch can run like this, BUT Elasticsearch will not run / will not run well if it can not claim the RAM it needs. So if 1gb of RAM works then fine. Also you can always start elasticsearch first let it claim its RAM then let the other processes "fight" of the remaining RAM.
So in this case you need to manually set the elasticsearch RAM as you just did... 1GB is pretty small... it may or may not work well depending on how many shards, ingest per sec, queries etc. you can always bump it up.
If it is not running well bump it up to 2 or 4 GB and start it first and so on.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.