We have a simple elasticsearch setup.
Yesterday the diskspace went full and I was not able to get into index management to delete old index.
So I deleted the files from disk.
Now elasticsearch wont start
I am running on a windows 2019 server
getting error
Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.kibana_task_manager_1][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open .apm-custom-link _aWvtxhsTgairMzpIjatHA 1 0
red open .kibana_task_manager_1 Msw7B2JjQZW2lYcyjk6BQw 1 0
red open winlogbeat-7.9.1-2020.11.17-000014 M5vFt0kcQGCt3eccwXEvGw 1 1
red open winlogbeat-7.9.1-2020.11.16-000013 -xYqiSsbQ9Sk_KjEQkiDqw 1 1
red open .apm-agent-configuration mJktuBDPTcGjXoVabrk0Lw 1 0
red open winlogbeat-7.9.1-2020.11.19-000016 3MJGeyIyS5GzEkOEN6FIJQ 1 1
red open winlogbeat-7.9.1-2020.11.18-000015 2asmAGBCR3GrRv052BU7Xw 1 1
red open winlogbeat-7.9.1-2020.11.15-000012 PmFFKQjyQ8anJdfe1Qg2zA 1 1
red open .kibana_1 8orVUZr9SjS-w7dRHoWTMw 1 0
I have new error
[2021-02-02T08:53:52,129][WARN ][r.suppressed ] [xxxxxx] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-02-02T08:53:53,705][WARN ][o.e.c.c.ClusterFormationFailureHelper] [xxxxxx] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{xxxxxxxxx}{xah9Pu-yRkWx-omoY99kCQ}{oWyNvwVHQTiXjn19gRDNzQ}{10.xx.xx.xx}{10.xx.xx.xx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [[fe80::ed9c:f27b:ea45:7a79]:9300] from hosts providers and [{xxxxxxxx}{xah9Pu-yRkWx-omoY99kCQ}{oWyNvwVHQTiXjn19gRDNzQ}{10.xx.xx.xx}{10.xx.xx.xx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
discovery.seed_hosts:
- xxxxxxxxx.xxxxxx.org:9300
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
cluster.routing.allocation.disk.threshold_enabled: false
[2021-02-02T10:58:18,672][WARN ][o.e.c.c.ClusterFormationFailureHelper] [XXXXXXXXXX] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{XXXXXXXXXX}{xah9Pu-yRkWx-omoY99kCQ}{El_3_k80QuGfd9-r4gDpHg}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [[::1]:9300] from hosts providers and [{XXXXXXXXXX}{xah9Pu-yRkWx-omoY99kCQ}{El_3_k80QuGfd9-r4gDpHg}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-02-02T10:58:20,469][WARN ][r.suppressed ] [XXXXXXXXXX] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-02-02T10:58:28,687][WARN ][o.e.c.c.ClusterFormationFailureHelper] [XXXXXXXXXX] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{XXXXXXXXXX}{xah9Pu-yRkWx-omoY99kCQ}{El_3_k80QuGfd9-r4gDpHg}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [[::1]:9300] from hosts providers and [{XXXXXXXXXX}{xah9Pu-yRkWx-omoY99kCQ}{El_3_k80QuGfd9-r4gDpHg}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}{dilmrt}{ml.machine_memory=34358718464, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-02-02T10:58:35,341][WARN ][r.suppressed ] [XXXXXXXXXX] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-02-02T10:58:35,341][WARN ][r.suppressed ] [XXXXXXXXXX] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.1.jar:7.9.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:651) [elasticsearch-7.9.1.jar:7.9.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
