# Elasticsearch.service craches (Active: failed) every 1,2 days **URL:** \ **Category:** Elasticsearch\ **Created:** [February 13, 2023, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373 "2023-02-13T10:24:14Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 13, 2023, 10:24am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/1 "2023-02-13T10:24:14Z")
Hi Team, every 1,2 days elasticsearch.service failed on my ubunto machine. It has all memory/disk resources it needs. I'll attach logs here and below is the failed status of elasticsearch. ============================================================ elasticsearch@seats-elk-vm:~$ sudo service elasticsearch status ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: signal) since Sat 2023-02-11 02:19:19 UTC; 2 days ago Docs: [https://www.elastic.co](https://www.elastic.co) Process: 1179 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID\_DIR}/elasticsearch.pid --quiet (code= Main PID: 1179 (code=killed, signal=KILL) Tasks: 0 (limit: 4673) CGroup: /system.slice/elasticsearch.service Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. lines 1-10/10 (END) ============================================================ Logs: > **[Logs.txt](https://drive.google.com/file/d/1yY-A2LdHkjxrd1JtRFKnHxfCsjd9_uAH/view?usp=share_link)** > > Google Drive file. Any help please ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 14, 2023, 4:54am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/2 "2023-02-14T04:54:29Z")
Welcome to our community! 😃 There's nothing in the Elasticsearch logs that would suggest why Elasticsearch would be stopping, is there anything in your OS logs? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 14, 2023, 5:19am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/3 "2023-02-14T05:19:32Z")
What kind of hardware is this node deployed on? How much resources does the node have assigned? It looks like the node may be starved of CPU resources as cluster state takes a long time to compute and timer threads frequently exceed sleep thresholds. ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 14, 2023, 9:15am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/4 "2023-02-14T09:15:52Z")
I don't think it could be the CPU resources as i just installed it and not used yet .. I needs to restart elasticsearch everyday in morning to get it back up .. Below are more logs from OS. elasticsearch@seats-elk-vm:~$ lsb\_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 18.04.6 LTS Release: 18.04 Codename: bionic elasticsearch@seats-elk-vm:~$ ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/48311d41b65054029ceb6ee65614201aa853de69.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/e/ce94cfd140e2fb836b2aa4bf4b8eefc26b5c495e.png) ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 14, 2023, 11:02am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/5 "2023-02-14T11:02:52Z")
elasticsearch@seats-elk-vm:/$ sudo service elasticsearch status ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled) Active: failed (Result: signal) since Mon 2023-02-13 15:04:25 UTC; 19h ago Docs: [https://www.elastic.co](https://www.elastic.co) Process: 9972 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID\_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL) Main PID: 9972 (code=killed, signal=KILL) Tasks: 0 (limit: 4673) CGroup: /system.slice/elasticsearch.service Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable. elasticsearch@seats-elk-vm:/$ ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [February 14, 2023, 11:11am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/6 "2023-02-14T11:11:02Z")
> [@Ziad\_Khater](#): > > I don't think it could be the CPU resources What are the resources of your server, You didn't share the exact number of CPUs and memory. Your systems seems to have only 4 GB of memory and you are running both Elasticsearch and Kibana on this, are you running anything else? You need to share the systems service, look at `/var/log/syslog` or `/var/log/messages` on the time interval that elasticsearch crashes. ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 14, 2023, 12:17pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/7 "2023-02-14T12:17:33Z")
> [@leandrojmp](#): > > /var/log/messages Hi @leandrojmp I've just attached all details in below files. Yes only will be used for Elasticsearch and Kibana. [Resources-packages.installed-Disk-Memory.txt - Google Drive](https://drive.google.com/file/d/1cmsx3V46UD3SkhScu4J-4FBuxJvOWi_H/view?usp=sharing), [syslog.txt - Google Drive](https://drive.google.com/file/d/1mFBppASZe7fddxyXPXrVTPpNvQm0bw78/view?usp=sharing), [syslog.2.txt - Google Drive](https://drive.google.com/file/d/1qcza4kDHDMglsA1mTvwaIwJ5oaWz4wH9/view?usp=sharing) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [February 14, 2023, 12:21pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/8 "2023-02-14T12:21:38Z")
There is nothing in the logs file that would indicate any issue related to Elasticsearch. You need to look if there is something about Elasticsearch in the system logs that could give some hint about the issue. When was the last time it failed? Look for the logs from that time. ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 14, 2023, 12:37pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/9 "2023-02-14T12:37:47Z")
Hi @leandrojmp , I find it fail every morning , Just found these logs below.. ```auto ][r.suppressed] [seats-elk-vm] path: /_security/_query/api_key, params: {pretty=true} org.elasticsearch.ElasticsearchException: Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node. ``` ```auto 0:33:43,638][INFO][o.e.p.PluginsService] [seats-elk-vm] no plugins loaded [2023-02-13T10:33:43,689][INFO][o.e.e.NodeEnvironment] [seats-elk-vm] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [23.1gb], net total_space [28.8gb], types [ext4] [2023-02-13T10:33:43,690][INFO][o.e.e.NodeEnvironment] [seats-elk-vm] heap size [1.9gb], compressed ordinary object pointers [true] [2023-02-13T10:33:43,853][INFO][o.e.n.Node] [seats-elk-vm] node name [seats-elk-vm], node ID [PZ4AAXiVRSmSWezIBzgZmg], cluster name [elasticsearch], roles [transform, data_frozen, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest] [2023-02-13T10:33:48,347][INFO][o.e.x.m.p.l.CppLogMessageHandler] [seats-elk-vm] [controller/10187] [Main.cc@122] controller (64 bit): Version 7.17.9 (Build ffceceeb3d63bc) Copyright (c) 2023 Elasticsearch BV [2023-02-13T10:33:49,447][INFO][o.e.x.s.a.s.FileRolesStore] [seats-elk-vm] parsed [0] roles from file [/etc/elasticsearch/roles.yml] [2023-02-13T10:33:50,077][INFO][o.e.i.g.ConfigDatabases] [seats-elk-vm] initialized default databases [[GeoLite2-Country.mmdb, GeoLite2-City.mmdb, GeoLite2-ASN.mmdb]], config databases [[]] and watching [/etc/elasticsearch/ingest-geoip] for changes [2023-02-13T10:33:50,078][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg] [2023-02-13T10:33:50,711][INFO][o.e.t.NettyAllocator] [seats-elk-vm] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}] [2023-02-13T10:33:50,745][INFO][o.e.i.r.RecoverySettings] [seats-elk-vm] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b] [2023-02-13T10:33:50,789][INFO][o.e.d.DiscoveryModule] [seats-elk-vm] using discovery type [zen] and seed hosts providers [settings] [2023-02-13T10:33:51,229][INFO][o.e.g.DanglingIndicesState] [seats-elk-vm] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually [2023-02-13T10:33:51,886][INFO][o.e.n.Node] [seats-elk-vm] initialized [2023-02-13T10:33:51,887][INFO][o.e.n.Node] [seats-elk-vm] starting ... [2023-02-13T10:33:51,912][INFO][o.e.x.s.c.f.PersistentCache] [seats-elk-vm] persistent cache index loaded [2023-02-13T10:33:51,913][INFO][o.e.x.d.l.DeprecationIndexingComponent] [seats-elk-vm] deprecation component started [2023-02-13T10:33:52,046][INFO][o.e.t.TransportService] [seats-elk-vm] publish_address {10.50.98.5:9300}, bound_addresses {[::]:9300} [2023-02-13T10:33:52,512][INFO][o.e.b.BootstrapChecks] [seats-elk-vm] bound or publishing to a non-loopback address, enforcing bootstrap checks [2023-02-13T10:33:52,538][INFO][o.e.c.c.Coordinator] [seats-elk-vm] cluster UUID [dwY8x9UwQ_uhhMGjSfFr8w] [2023-02-13T10:33:52,681][INFO][o.e.c.s.MasterService] [seats-elk-vm] elected-as-master ([1] nodes joined)[{seats-elk-vm}{PZ4AAXiVRSmSWezIBzgZmg}{oV1QVwS-QHG-u1aokRv4og}{10.50.98.5}{10.50.98.5:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 7, version: 154, delta: master node changed {previous [], current [{seats-elk-vm}{PZ4AAXiVRSmSWezIBzgZmg}{oV1QVwS-QHG-u1aokRv4og}{10.50.98.5}{10.50.98.5:9300}{cdfhilmrstw}]} [2023-02-13T10:33:52,798][INFO][o.e.c.s.ClusterApplierService] [seats-elk-vm] master node changed {previous [], current [{seats-elk-vm}{PZ4AAXiVRSmSWezIBzgZmg}{oV1QVwS-QHG-u1aokRv4og}{10.50.98.5}{10.50.98.5:9300}{cdfhilmrstw}]}, term: 7, version: 154, reason: Publication{term=7, version=154} [2023-02-13T10:33:52,859][INFO][o.e.h.AbstractHttpServerTransport] [seats-elk-vm] publish_address {10.50.98.5:9200}, bound_addresses {[::]:9200} [2023-02-13T10:33:52,860][INFO][o.e.n.Node] [seats-elk-vm] started [2023-02-13T10:33:53,285][INFO][o.e.l.LicenseService] [seats-elk-vm] license [bb80596b-6f22-45e7-a429-672184e91914] mode [basic] - valid [2023-02-13T10:33:53,286][INFO][o.e.x.s.s.SecurityStatusChangeListener] [seats-elk-vm] Active license is now [BASIC]; Security is disabled [2023-02-13T10:33:53,287][WARN][o.e.x.s.s.SecurityStatusChangeListener] [seats-elk-vm] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security. [2023-02-13T10:33:53,294][INFO][o.e.g.GatewayService] [seats-elk-vm] recovered [11] indices into cluster_state [2023-02-13T10:33:54,055][ERROR][o.e.i.g.GeoIpDownloader] [seats-elk-vm] exception during geoip databases update org.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active at org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:137) ~[ingest-geoip-7.17.9.jar:7.17.9] at org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:284) [ingest-geoip-7.17.9.jar:7.17.9] at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:100) [ingest-geoip-7.17.9.jar:7.17.9] at org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:46) [ingest-geoip-7.17.9.jar:7.17.9] at org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.9.jar:7.17.9] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?] at java.lang.Thread.run(Thread.java:1589) [?:?] [2023-02-13T10:33:54,485][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] retrieve geoip database [GeoLite2-Country.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-Country.mmdb.tmp.gz] [2023-02-13T10:33:54,486][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] retrieve geoip database [GeoLite2-City.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-City.mmdb.tmp.gz] [2023-02-13T10:33:54,488][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] retrieve geoip database [GeoLite2-ASN.mmdb] from [.geoip_databases] to [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-ASN.mmdb.tmp.gz] [2023-02-13T10:33:55,237][INFO][o.e.c.r.a.AllocationService] [seats-elk-vm] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.ds-.logs-deprecation.elasticsearch-default-2023.02.08-000001][0]]]). [2023-02-13T10:33:55,360][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] successfully reloaded changed geoip database file [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-Country.mmdb] [2023-02-13T10:33:55,819][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] successfully reloaded changed geoip database file [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-ASN.mmdb] [2023-02-13T10:33:56,995][INFO][o.e.i.g.DatabaseNodeService] [seats-elk-vm] successfully reloaded changed geoip database file [/tmp/elasticsearch-1983527832423388062/geoip-databases/PZ4AAXiVRSmSWezIBzgZmg/GeoLite2-City.mmdb] [2023-02-13T12:03:00,518][INFO][o.e.t.LoggingTaskListener] [seats-elk-vm] 58943 finished with response BulkByScrollResponse[took=74.5ms,timed_out=false,sliceId=null,updated=17,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]] [2023-02-13T12:03:00,587][INFO][o.e.t.LoggingTaskListener] [seats-elk-vm] 58944 finished with response BulkByScrollResponse[took=300.6ms,timed_out=false,sliceId=null,updated=342,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]] [2023-02-13T12:03:02,794][INFO][o.e.x.i.a.TransportPutLifecycleAction] [seats-elk-vm] updating index lifecycle policy [.alerts-ilm-policy] [2023-02-13T12:24:06,665][INFO][o.e.t.LoggingTaskListener] [seats-elk-vm] 74860 finished with response BulkByScrollResponse[took=40.6ms,timed_out=false,sliceId=null,updated=17,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]] [2023-02-13T12:24:06,897][INFO][o.e.t.LoggingTaskListener] [seats-elk-vm] 74883 finished with response BulkByScrollResponse[took=171.5ms,timed_out=false,sliceId=null,updated=348,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]] [2023-02-13T12:24:09,068][INFO][o.e.x.i.a.TransportPutLifecycleAction] [seats-elk-vm] updating index lifecycle policy [.alerts-ilm-policy] [2023-02-13T12:26:09,407][WARN][r.suppressed] [seats-elk-vm] path: /_security/_query/api_key, params: {pretty=true} org.elasticsearch.ElasticsearchException: Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node. at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.checkFeatureAvailable(SecurityBaseRestHandler.java:73) ~[x-pack-security-7.17.9.jar:7.17.9] at org.elasticsearch.xpack.security.rest.action.SecurityBaseRestHandler.prepareRequest(SecurityBaseRestHandler.java:49) [x-pack-security-7.17.9.jar:7.17.9] at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:86) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:105) [x-pack-security-7.17.9.jar:7.17.9] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:337) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:403) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:255) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:382) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:461) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:357) [elasticsearch-7.17.9.jar:7.17.9] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:35) [transport-netty4-client-7.17.9.jar:7.17.9] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:19) [transport-netty4-client-7.17.9.jar:7.17.9] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:48) [transport-netty4-client-7.17.9.jar:7.17.9] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.66.Final.jar:4.1.66.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.66.Final.jar:4.1.66.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.66.Final.jar:4.1.66.Final] at java.lang.Thread.run(Thread.java:1589) [?:?] [2023-02-13T14:40:10,742][WARN][o.e.t.ThreadPool] [seats-elk-vm] execution of [ReschedulingRunnable{runnable=org.elasticsearch.watcher.ResourceWatcherService$ResourceMonitor@5d96df62, interval=5s}] took [206555ms] which is above the warn threshold of [5000ms] [2023-02-13T14:40:10,996][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [16s/16097ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:42:23,997][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [16s/16096630100ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:43:15,504][WARN][o.e.t.ThreadPool] [seats-elk-vm] execution of [ReschedulingRunnable{runnable=org.elasticsearch.monitor.jvm.JvmGcMonitorService$1@3cedb60, interval=1s}] took [226345ms] which is above the warn threshold of [5000ms] [2023-02-13T14:43:40,746][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [3.7m/226345ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:43:55,957][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [3.7m/226345452400ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:44:17,554][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.5m/91515ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:44:37,447][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.5m/91514485300ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:44:52,508][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [38.8s/38838ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:45:12,206][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [38.8s/38838241200ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:45:42,348][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [49.9s/49952ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:46:05,346][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [49.9s/49952197800ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:46:01,802][WARN][o.e.t.ThreadPool] [seats-elk-vm] execution of [ReschedulingRunnable{runnable=org.elasticsearch.search.SearchService$Reaper@482ec1f9, interval=1m}] took [49952ms] which is above the warn threshold of [5000ms] [2023-02-13T14:46:26,448][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [44.5s/44550ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:46:38,501][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [44.5s/44550112600ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:47:02,046][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [35.2s/35247ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:47:16,202][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [35.2s/35246191600ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:47:53,345][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [51.4s/51462ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:47:34,695][WARN][o.e.m.f.FsHealthService] [seats-elk-vm] health check of [/var/lib/elasticsearch/nodes/0] took [35246ms] which is above the warn threshold of [5s] [2023-02-13T14:48:40,105][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [51.4s/51462090800ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:50:10,396][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [2.2m/136650ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:50:38,002][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [2.2m/136650183800ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:51:12,002][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1m/61544ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:52:11,002][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1m/61544485400ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:52:31,305][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.3m/79253ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:54:15,544][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.3m/79252940400ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:54:42,850][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [2.1m/131551ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:55:32,097][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [2.1m/131550906400ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:56:27,855][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.7m/103703ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:56:57,996][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.7m/103702725100ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:57:30,393][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1m/63996ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T14:58:27,752][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1m/63995692200ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T14:59:03,446][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.5m/93103ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T15:00:05,946][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.5m/93103457100ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T15:01:00,500][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.9m/116650ms] on absolute clock which is above the warn threshold of [5000ms] [2023-02-13T15:01:43,507][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.9m/116650064800ns] on relative clock which is above the warn threshold of [5000ms] [2023-02-13T15:02:18,555][WARN][o.e.t.ThreadPool] [seats-elk-vm] timer thread slept for [1.2m/77236ms] on absolute clock which is above the warn threshold of [5000ms] elasticsearch@seats-elk-vm:/var/log/elasticsearch$ ``` I guess it has something with security level .. ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 14, 2023, 3:19pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/10 "2023-02-14T15:19:30Z")
> [@Ziad\_Khater](#): > > `security.enabled` Tried to add ```auto xpack.security.enabled: true ``` to /etc/elasticsearch/elasticsearch.yml and restart elasticsearch service but it failed.. Any more ideas ? ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 15, 2023, 8:58am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/11 "2023-02-15T08:58:55Z")
```auto elasticsearch@seats-elk-vm:~$ curl http://127.0.0.1:9200/_security/_authenticate?pretty { "error" : { "root_cause" : [ { "type" : "exception", "reason" : "Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node." } ], "type" : "exception", "reason" : "Security must be explicitly enabled when using a [basic] license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node." }, "status" : 500 } ``` My attempt to solve it 1- ```auto sudo nano /etc/elasticsearch/elasticsearch.yml ``` ```auto xpack.security.enabled: true ``` ```auto sudo systemctl restart elasticsearch ``` ```auto root@seats-elk-vm:/home/elasticsearch# sudo systemctl start elasticsearch Job for elasticsearch.service failed because the control process exited with error code. See "systemctl status elasticsearch.service" and "journalctl -xe" for details. root@seats-elk-vm:/home/elasticsearch# sudo nano /etc/elasticsearch/elasticsearch.yml ``` When i remove ```auto xpack.security.enabled: true ``` it gets back online .. Any more Suggestions ? ---
**Author:** ![Ziad\_Khater](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ziad_khater/32/116958_2.png) [@Ziad\_Khater](https://discuss.elastic.co/u/Ziad_Khater)\ **Post date:** [February 15, 2023, 10:32am UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/12 "2023-02-15T10:32:45Z")
@Christian_Dahlqvist @warkolm @leandrojmp ? 🙂 ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [February 15, 2023, 12:42pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/13 "2023-02-15T12:42:09Z")
Again, there is nothing in your log that would indicate the reason it is failing, you need to look for WARN and ERROR lines **at the moment** it stopped, what you shared has no information about it, it just shows that your node started but you have some configuration issue regarding security. It seems that you didn't configure security for your node, is that right? If you didn't configure security for your node you need to configure it following the documentation or set the `xpack.security.enabled` to `false` (not recommended). What do you have in the logs after you explicitly enable security? You didn't shared, you need to look both in the systems log and Elasticsearch log, showing that the service failed does not help, you need to share the log of why it failed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 15, 2023, 12:42pm UTC](https://discuss.elastic.co/t/elasticsearch-service-craches-active-failed-every-1-2-days/325373/14 "2023-03-15T12:42:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.