My Syslog message is: Out of memory: Killed process 1421927 (java) total-vm:17109192kB, anon-rss:6968532kB, file-rss:0kB, shmem-rss:0kB, UID:128 pgtables:18984kB oom_score_adj
In which file i need to define memory:
-rw-rw---- 1 root elasticsearch 0 Jun 23 2023 users_roles
-rw-rw---- 1 root elasticsearch 0 Jun 23 2023 users
-rw-rw---- 1 root elasticsearch 197 Jun 23 2023 roles.yml
-rw-rw---- 1 root elasticsearch 473 Jun 23 2023 role_mapping.yml
-rw-rw---- 1 root elasticsearch 19304 Jun 23 2023 log4j2.properties
drwxr-s--- 2 root elasticsearch 4096 Jun 23 2023 jvm.options.d
-rw-rw---- 1 root elasticsearch 3329 Jun 23 2023 jvm.options
-rw-rw---- 1 root elasticsearch 1042 Jun 23 2023 elasticsearch-plugins.example.yml
-rw-rw---- 1 root elasticsearch 199 Jul 2 2023 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch 3458 May 27 10:21 elasticsearch.yml
root@lmcuser:/etc/elasticsearch# free -m
root@lmcuser:/etc/elasticsearch# swapon -s
As specified in the documentation Elasticsearch should always be run with swapping disabled as it can affect both performance and stability.
Please address this and see if that resolves your issue.
Ok i will try and update You! thanx for the prompt reply! 
i have disabled swapping as per the share documents but still my elasticsearch service got killed around 12:50 PM IST
Syslog Error Message is -->
May 29 12:51:09 lmcuser kernel: [3815630.759755] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/elasticsearch.service,task=java,pid=31380>
May 29 12:51:09 lmcuser kernel: [3815630.759899] Out of memory: Killed process 3138075 (java) total-vm:17134548kB, anon-rss:9159316kB, file-rss:0kB, shmem-rss:0kB, UID:128 pgtables:19620kB oom_score_adj>
May 29 12:51:09 lmcuser dbus-daemon[3801]: [session uid=1001 pid=3801] Successfully activated service 'org.freedesktop.Tracker1.Miner.Extract'
May 29 12:51:09 lmcuser systemd[3786]: Started Tracker metadata extractor.
May 29 12:51:09 lmcuser systemd[1]: elasticsearch.service: Main process exited, code=killed, status=9/KILL
May 29 12:51:09 lmcuser systemd[1]: elasticsearch.service: Failed with result 'signal'.
what are the other settings i have to check?
Is there anything else running on that host?
What is the full output of the cluster stats API?
What is in the Elasticsearch logs (location depends on how you installed it)?
Output of cluster stats APi is:
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "gmrtdell7000",
"cluster_uuid" : "4UOYQ-HfRB6NdUlve4CoUQ",
"timestamp" : 1716978123114,
"status" : "yellow",
"indices" : {
"count" : 50,
"shards" : {
"total" : 50,
"primaries" : 50,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"primaries" : {
"min" : 1,
"max" : 1,
"avg" : 1.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 3472237,
"deleted" : 762470
},
"store" : {
"size_in_bytes" : 1616930410,
"total_data_set_size_in_bytes" : 1616930410,
"reserved_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 0,
"evictions" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 1400,
"total_count" : 187,
"hit_count" : 151,
"miss_count" : 36,
"cache_size" : 4,
"cache_count" : 4,
"evictions" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 215,
"memory_in_bytes" : 2231252,
"terms_memory_in_bytes" : 869704,
"stored_fields_memory_in_bytes" : 107928,
"term_vectors_memory_in_bytes" : 0,
"norms_memory_in_bytes" : 56064,
"points_memory_in_bytes" : 0,
"doc_values_memory_in_bytes" : 1197556,
"index_writer_memory_in_bytes" : 11107842,
"version_map_memory_in_bytes" : 847782,
"fixed_bit_set_memory_in_bytes" : 330992,
"max_unsafe_auto_id_timestamp" : 1716976528051,
"file_sizes" : { }
},
"mappings" : {
"field_types" : [
{
"name" : "alias",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "binary",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "boolean",
"count" : 61,
"index_count" : 24,
"script_count" : 0
},
{
"name" : "byte",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "constant_keyword",
"count" : 10,
"index_count" : 4,
"script_count" : 0
},
{
"name" : "date",
"count" : 138,
"index_count" : 39,
"script_count" : 0
},
{
"name" : "date_nanos",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "date_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "double",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "double_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "float",
"count" : 450,
"index_count" : 19,
"script_count" : 0
},
{
"name" : "float_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "geo_point",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "geo_shape",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "half_float",
"count" : 65,
"index_count" : 17,
"script_count" : 0
},
{
"name" : "integer",
"count" : 180,
"index_count" : 10,
"script_count" : 0
},
{
"name" : "integer_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "ip",
"count" : 5,
"index_count" : 5,
"script_count" : 0
},
{
"name" : "ip_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "keyword",
"count" : 999,
"index_count" : 39,
"script_count" : 0
},
{
"name" : "long",
"count" : 1501,
"index_count" : 31,
"script_count" : 0
},
{
"name" : "long_range",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "nested",
"count" : 28,
"index_count" : 12,
"script_count" : 0
},
{
"name" : "object",
"count" : 1026,
"index_count" : 30,
"script_count" : 0
},
{
"name" : "shape",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "short",
"count" : 1,
"index_count" : 1,
"script_count" : 0
},
{
"name" : "text",
"count" : 216,
"index_count" : 30,
"script_count" : 0
},
{
"name" : "version",
"count" : 4,
"index_count" : 4,
"script_count" : 0
}
],
"runtime_field_types" : [ ]
},
"analysis" : {
"char_filter_types" : [ ],
"tokenizer_types" : [ ],
"filter_types" : [ ],
"analyzer_types" : [ ],
"built_in_char_filters" : [ ],
"built_in_tokenizers" : [ ],
"built_in_filters" : [ ],
"built_in_analyzers" : [ ]
},
"versions" : [
{
"version" : "7.17.11",
"index_count" : 50,
"primary_shard_count" : 50,
"total_primary_bytes" : 1616930410
}
]
},
"nodes" : {
"count" : {
"total" : 1,
"coordinating_only" : 0,
"data" : 1,
"data_cold" : 1,
"data_content" : 1,
"data_frozen" : 1,
"data_hot" : 1,
"data_warm" : 1,
"ingest" : 1,
"master" : 1,
"ml" : 1,
"remote_cluster_client" : 1,
"transform" : 1,
"voting_only" : 0
},
"versions" : [
"7.17.11"
],
"os" : {
"available_processors" : 12,
"allocated_processors" : 12,
"names" : [
{
"name" : "Linux",
"count" : 1
}
],
"pretty_names" : [
{
"pretty_name" : "Ubuntu 20.04.6 LTS",
"count" : 1
}
],
"architectures" : [
{
"arch" : "amd64",
"count" : 1
}
],
"mem" : {
"total_in_bytes" : 16440324096,
"free_in_bytes" : 4301279232,
"used_in_bytes" : 12139044864,
"free_percent" : 26,
"used_percent" : 74
}
},
"process" : {
"cpu" : {
"percent" : 0
},
"open_file_descriptors" : {
"min" : 627,
"max" : 627,
"avg" : 627
}
},
"jvm" : {
"max_uptime_in_millis" : 1496424,
"versions" : [
{
"version" : "20.0.1",
"vm_name" : "OpenJDK 64-Bit Server VM",
"vm_version" : "20.0.1+9-29",
"vm_vendor" : "Oracle Corporation",
"bundled_jdk" : true,
"using_bundled_jdk" : true,
"count" : 1
}
],
"mem" : {
"heap_used_in_bytes" : 895433728,
"heap_max_in_bytes" : 8220835840
},
"threads" : 118
},
"fs" : {
"total_in_bytes" : 241569767424,
"free_in_bytes" : 159334412288,
"available_in_bytes" : 146988740608
},
"plugins" : [ ],
"network_types" : {
"transport_types" : {
"security4" : 1
},
"http_types" : {
"security4" : 1
}
},
"discovery_types" : {
"zen" : 1
},
"packaging_types" : [
{
"flavor" : "default",
"type" : "deb",
"count" : 1
}
],
"ingest" : {
"number_of_pipelines" : 21,
"processor_stats" : {
"conditional" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"convert" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"geoip" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"grok" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"gsub" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"remove" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"rename" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"script" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"set" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
},
"set_security_user" : {
"count" : 0,
"failed" : 0,
"current" : 0,
"time_in_millis" : 0
}
}
}
}
}
==>
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$4(SecurityActionFilter.java:186) [x-pack-security-7.17.11.jar:7.17.11]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.security.authc.AuthenticatorChain.authenticateAsync(AuthenticatorChain.java:102) [x-pack-security-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:171) [x-pack-security-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:182) [x-pack-security-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:124) [x-pack-security-7.17.11.jar:7.17.11]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:184) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:161) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:55) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) [elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:251) [x-pack-core-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:211) [x-pack-core-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.transform.checkpoint.TimeBasedCheckpointProvider.sourceHasChanged(TimeBasedCheckpointProvider.java:80) [transform-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.transform.transforms.TransformIndexer.sourceHasChanged(TransformIndexer.java:1017) [transform-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.transform.transforms.TransformIndexer.onStart(TransformIndexer.java:367) [transform-7.17.11.jar:7.17.11]
at org.elasticsearch.xpack.core.indexing.AsyncTwoPhaseIndexer.lambda$maybeTriggerAsyncJob$5(AsyncTwoPhaseIndexer.java:219) [x-pack-core-7.17.11.jar:7.17.11]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.11.jar:7.17.11]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.lang.Thread.run(Thread.java:1623) [?:?]
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider using `allow_partial_search_res>
at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:227) ~[elasticsearch-7.17.11.jar:7.17.11]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:454) [elasticsearch-7.17.11.jar:7.17.11]
... 90 more
[2024-05-29T15:25:27,989][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-5801097677920706396/geoip-databases>
[2024-05-29T15:25:27,989][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-5801097677920706396/geoip-databases/Vo>
[2024-05-29T15:25:27,990][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-5801097677920706396/geoip-databases/VoM>
[2024-05-29T15:25:28,051][INFO ][o.e.x.t.t.TransformPersistentTasksExecutor] [lmcuser] [endpoint.metadata_current-default-1.3.0] successfully completed and scheduled task in node operation
[2024-05-29T15:25:28,104][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] successfully reloaded changed geoip database file [/tmp/elasticsearch-5801097677920706396/geoip-databases/VoMIuH2dRXSmPOzVaUFXfw/G>
[2024-05-29T15:25:28,140][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] successfully reloaded changed geoip database file [/tmp/elasticsearch-5801097677920706396/geoip-databases/VoMIuH2dRXSmPOzVaUFXfw/G>
[2024-05-29T15:25:28,582][INFO ][o.e.i.g.DatabaseNodeService] [lmcuser] successfully reloaded changed geoip database file [/tmp/elasticsearch-5801097677920706396/geoip-databases/VoMIuH2dRXSmPOzVaUFXfw/G>
[2024-05-29T15:25:30,212][INFO ][o.e.t.LoggingTaskListener] [lmcuser] 504 finished with response BulkByScrollResponse[took=40.8ms,timed_out=false,sliceId=null,updated=18,created=0,deleted=0,batches=1,ve>
[2024-05-29T15:25:31,067][INFO ][o.e.t.LoggingTaskListener] [lmcuser] 503 finished with response BulkByScrollResponse[took=900.8ms,timed_out=false,sliceId=null,updated=3401,created=0,deleted=0,batches=4>
[2024-05-29T15:25:32,240][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [lmcuser] updating index lifecycle policy [.alerts-ilm-policy]
[2024-05-29T15:25:34,607][INFO ][o.e.c.r.a.AllocationService] [lmcuser] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.monitoring-es-7-2024.05.29][0]]]).
[2024-05-29T15:47:05,358][WARN ][o.e.t.ThreadPool ] [lmcuser] timer thread slept for [1.7m/107218ms] on absolute clock which is above the warn threshold of [5000ms]
My logs location is: is it correct :
-rw-r--r-- 1 elasticsearch elasticsearch 30761 May 27 09:00 gmrtdell7000-2024-05-26-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 340148 May 27 10:23 gc.log.04
-rw-r--r-- 1 elasticsearch elasticsearch 2179 May 27 10:23 gc.log.05
-rw-r--r-- 1 elasticsearch elasticsearch 12446 May 28 01:00 gmrtdell7000-2024-05-27-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 13358 May 28 01:00 gmrtdell7000-2024-05-27-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch 2998827 May 28 22:45 gc.log.06
-rw-r--r-- 1 elasticsearch elasticsearch 2179 May 29 09:42 gc.log.07
-rw-r--r-- 1 elasticsearch elasticsearch 965 May 29 09:42 gmrtdell7000-2024-05-28-1.json.gz
-rw-r--r-- 1 elasticsearch elasticsearch 795 May 29 09:42 gmrtdell7000-2024-05-28-1.log.gz
-rw-r--r-- 1 elasticsearch elasticsearch 25016 May 29 09:42 gc.log.08
-rw-r--r-- 1 elasticsearch elasticsearch 2179 May 29 09:44 gc.log.09
-rw-r--r-- 1 elasticsearch elasticsearch 476709 May 29 12:50 gc.log.10
-rw-r--r-- 1 elasticsearch elasticsearch 2104 May 29 15:25 gc.log.11
-rw-r--r-- 1 elasticsearch elasticsearch 690882 May 29 15:47 gmrtdell7000_server.json
-rw-r--r-- 1 elasticsearch elasticsearch 630411 May 29 15:47 gmrtdell7000.log
-rw-r--r-- 1 elasticsearch elasticsearch 28798 May 29 15:50 gmrtdell7000_deprecation.log
-rw-r--r-- 1 elasticsearch elasticsearch 59516 May 29 15:50 gmrtdell7000_deprecation.json
-rw-r--r-- 1 elasticsearch elasticsearch 234270 May 29 16:04 gc.log
root@lmcuser:/var/log/elasticsearch#
You missed this question.
i have installed Elasticsearch on Single machine (Single node). On that machine Cronjob is running which send data to elasticsearch index every 1 minute.
no any other things is running on Host machine.