Setting up elastic cluster

elastic_dev · September 8, 2021, 2:20pm

Hi, I am setting up the Elasticsearch with 2 nodes. whenever I start my node-2 it gives me error as below

'[2021-09-08T19:24:43,093][ERROR][o.e.i.g.DatabaseRegistry ] [node-2] failed to download database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:42) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:358) [ingest-geoip-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.1.jar:7.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] '

However status of cluster shows green

{
"cluster_name" : "my-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Anyone, please help me why I am getting above error

warkolm · September 9, 2021, 12:05am

Welcome to our community!
Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

Can you please post your Elasticsearch log, from startup to the latest entry. If you cannot post it here, please use gist/pastebin/etc.

elastic_dev · September 9, 2021, 5:25am

Hi Below are the Elasticsearch logs

[2021-09-09T10:39:42,472][INFO ][o.e.i.g.DatabaseRegistry ] [node-3] initialized database registry, using geoip-databases directory [/tmp/elasticsearch-6734919258144582330/geoip-databases/emjzzY5FQRKIDuhd-qAEvg]
[2021-09-09T10:39:46,438][INFO ][o.e.t.NettyAllocator     ] [node-3] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-09-09T10:39:46,792][INFO ][o.e.d.DiscoveryModule    ] [node-3] using discovery type [zen] and seed hosts providers [settings]
[2021-09-09T10:39:48,989][INFO ][o.e.g.DanglingIndicesState] [node-3] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-09-09T10:39:50,681][INFO ][o.e.n.Node               ] [node-3] initialized
[2021-09-09T10:39:50,691][INFO ][o.e.n.Node               ] [node-3] starting ...
[2021-09-09T10:39:51,451][INFO ][o.e.x.s.c.f.PersistentCache] [node-3] persistent cache index loaded
[2021-09-09T10:39:53,930][INFO ][o.e.t.TransportService   ] [node-3] publish_address {192.168.29.48:9300}, bound_addresses {[::]:9300}
[2021-09-09T10:39:57,173][INFO ][o.e.b.BootstrapChecks    ] [node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-09T10:39:57,227][INFO ][o.e.c.c.Coordinator      ] [node-3] cluster UUID [ZXN5Tg1GSwaPNgQuIcE9ZQ]
[2021-09-09T10:40:00,136][INFO ][o.e.c.s.ClusterApplierService] [node-3] master node changed {previous [], current [{node-2}{s2FQVvmDRKSqS_yYCOnWxA}{xFl8XFzKSTaGTn4bOM4gDA}{192.168.29.182}{192.168.29.182:9300}{cdfhilmrstw}]}, added {{node-1}{L_MU8MKAQhyBuMQYgkhQhA}{EV2ZU5ZmTG-NrNk_Eptcnw}{192.168.29.40}{192.168.29.40:9300}{cdfhilmrstw}, {node-2}{s2FQVvmDRKSqS_yYCOnWxA}{xFl8XFzKSTaGTn4bOM4gDA}{192.168.29.182}{192.168.29.182:9300}{cdfhilmrstw}}, term: 5, version: 113, reason: ApplyCommitRequest{term=5, version=113, sourceNode={node-2}{s2FQVvmDRKSqS_yYCOnWxA}{xFl8XFzKSTaGTn4bOM4gDA}{192.168.29.182}{192.168.29.182:9300}{cdfhilmrstw}{ml.machine_memory=16716709888, ml.max_open_jobs=512, xpack.installed=true, ml.max_jvm_size=8359247872, transform.node=true}}
[2021-09-09T10:40:00,798][INFO ][o.e.i.g.DatabaseRegistry ] [node-3] downloading geoip database [GeoLite2-Country.mmdb] to [/tmp/elasticsearch-6734919258144582330/geoip-databases/emjzzY5FQRKIDuhd-qAEvg/GeoLite2-Country.mmdb.tmp.gz]
[2021-09-09T10:40:00,802][INFO ][o.e.i.g.DatabaseRegistry ] [node-3] downloading geoip database [GeoLite2-City.mmdb] to [/tmp/elasticsearch-6734919258144582330/geoip-databases/emjzzY5FQRKIDuhd-qAEvg/GeoLite2-City.mmdb.tmp.gz]
[2021-09-09T10:40:00,803][INFO ][o.e.i.g.DatabaseRegistry ] [node-3] downloading geoip database [GeoLite2-ASN.mmdb] to [/tmp/elasticsearch-6734919258144582330/geoip-databases/emjzzY5FQRKIDuhd-qAEvg/GeoLite2-ASN.mmdb.tmp.gz]
[2021-09-09T10:40:00,965][ERROR][o.e.i.g.DatabaseRegistry ] [node-3] failed to download database [GeoLite2-ASN.mmdb]
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:605) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:494) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$3(TransportSearchAction.java:288) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:134) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:103) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:76) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:329) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:217) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:93) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:122) ~[?:?]
	at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:171) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:149) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:77) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:66) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:40) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:402) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:390) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:534) ~[elasticsearch-7.14.1.jar:7.14.1]
	at org.elasticsearch.ingest.geoip.DatabaseRegistry.lambda$retrieveDatabase$11(DatabaseRegistry.java:358) [ingest-geoip-7.14.1.jar:7.14.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.1.jar:7.14.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:831) [?:?]
[2021-09-09T10:40:02,987][INFO ][o.e.x.s.a.TokenService   ] [node-3] refresh keys
[2021-09-09T10:40:03,492][INFO ][o.e.x.s.a.TokenService   ] [node-3] refreshed keys
[2021-09-09T10:40:03,706][INFO ][o.e.l.LicenseService     ] [node-3] license [6d1b0a1f-57b4-4c1d-a49f-fae323fdc71f] mode [basic] - valid
[2021-09-09T10:40:03,715][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-3] Active license is now [BASIC]; Security is disabled
[2021-09-09T10:40:03,716][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [node-3] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.14/security-minimal-setup.html to enable security.
[2021-09-09T10:40:04,821][INFO ][o.e.h.AbstractHttpServerTransport] [node-3] publish_address {192.168.29.48:9200}, bound_addresses {[::]:9200}
[2021-09-09T10:40:04,823][INFO ][o.e.n.Node               ] [node-3] started

system · October 7, 2021, 5:25am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.