Elasticsearch install error on MAC

Hi,

I am new to Elasticsearch, tried to install, when I try to hit the URL "http://localhost:9200/", it works fine. However when I need to reset a password it gives an error.

./elasticsearch-reset-password -u kibana_system
Failed to determine the health of the cluster. Cluster health is currently RED.
This means that some cluster data is unavailable and your cluster is not fully functional.
The cluster logs (https://www.elastic.co/guide/en/elasticsearch/reference/8.5/logging.html) might contain information/indications for the underlying cause
It is recommended that you resolve the issues with your cluster before continuing
It is very likely that the command will fail when run against an unhealthy cluster.

If you still want to attempt to execute this command against an unhealthy cluster, you can pass the `-f` parameter.

ERROR: Failed to determine the health of the cluster. Cluster health is currently RED.

Logs

[2022-12-20T21:59:18,921][WARN ][stderr                   ] [Shanths-MBP-15.lan] WARNING: You are running with Java 19. To make full use of MMapDirectory, please pass '--enable-preview' to the Java command line.
[2022-12-20T21:59:18,933][INFO ][o.e.e.NodeEnvironment    ] [Shanths-MBP-15.lan] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [37.2gb], net total_space [931.5gb], types [apfs]
[2022-12-20T21:59:18,933][INFO ][o.e.e.NodeEnvironment    ] [Shanths-MBP-15.lan] heap size [16gb], compressed ordinary object pointers [true]
[2022-12-20T21:59:18,989][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] node name [Shanths-MBP-15.lan], node ID [Gqqcnr5GRoKOol-P38vgrQ], cluster name [elasticsearch], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]
[2022-12-20T21:59:21,689][INFO ][o.e.x.s.Security         ] [Shanths-MBP-15.lan] Security is disabled
[2022-12-20T21:59:21,779][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Shanths-MBP-15.lan] [controller/76924] [Main.cc@123] controller (64 bit): Version 8.5.3 (Build 32c68d5e2f9f8f) Copyright (c) 2022 Elasticsearch BV
[2022-12-20T21:59:22,348][INFO ][o.e.t.n.NettyAllocator   ] [Shanths-MBP-15.lan] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2022-12-20T21:59:22,369][INFO ][o.e.i.r.RecoverySettings ] [Shanths-MBP-15.lan] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-12-20T21:59:22,407][INFO ][o.e.d.DiscoveryModule    ] [Shanths-MBP-15.lan] using discovery type [multi-node] and seed hosts providers [settings]
[2022-12-20T21:59:23,397][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] initialized
[2022-12-20T21:59:23,398][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] starting ...
[2022-12-20T21:59:23,472][INFO ][o.e.x.s.c.f.PersistentCache] [Shanths-MBP-15.lan] persistent cache index loaded
[2022-12-20T21:59:23,473][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [Shanths-MBP-15.lan] deprecation component started
[2022-12-20T21:59:23,831][INFO ][o.e.t.TransportService   ] [Shanths-MBP-15.lan] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2022-12-20T21:59:24,358][WARN ][o.e.c.c.ClusterBootstrapService] [Shanths-MBP-15.lan] this node is locked into cluster UUID [pc38WIR0RmuJRnkKkidQUw] but [cluster.initial_master_nodes] is set to [Shanths-MBP-15.lan]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts
[2022-12-20T21:59:24,519][INFO ][o.e.c.s.MasterService    ] [Shanths-MBP-15.lan] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {Shanths-MBP-15.lan}{Gqqcnr5GRoKOol-P38vgrQ}{oMBLI8KXQRy_-KQOGDaQKw}{Shanths-MBP-15.lan}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw} completing election], term: 9, version: 69, delta: master node changed {previous [], current [{Shanths-MBP-15.lan}{Gqqcnr5GRoKOol-P38vgrQ}{oMBLI8KXQRy_-KQOGDaQKw}{Shanths-MBP-15.lan}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}
[2022-12-20T21:59:24,616][INFO ][o.e.c.s.ClusterApplierService] [Shanths-MBP-15.lan] master node changed {previous [], current [{Shanths-MBP-15.lan}{Gqqcnr5GRoKOol-P38vgrQ}{oMBLI8KXQRy_-KQOGDaQKw}{Shanths-MBP-15.lan}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 9, version: 69, reason: Publication{term=9, version=69}
[2022-12-20T21:59:24,635][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] flood stage disk watermark [95%] exceeded on [Gqqcnr5GRoKOol-P38vgrQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 37.2gb[3.9%], all indices on this node will be marked read-only
[2022-12-20T21:59:24,647][INFO ][o.e.r.s.FileSettingsService] [Shanths-MBP-15.lan] starting file settings watcher ...
[2022-12-20T21:59:24,651][INFO ][o.e.r.s.FileSettingsService] [Shanths-MBP-15.lan] file settings service up and running [tid=85]
[2022-12-20T21:59:24,657][INFO ][o.e.h.AbstractHttpServerTransport] [Shanths-MBP-15.lan] publish_address {192.168.10.132:9200}, bound_addresses {[::]:9200}
[2022-12-20T21:59:24,658][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] started {Shanths-MBP-15.lan}{Gqqcnr5GRoKOol-P38vgrQ}{oMBLI8KXQRy_-KQOGDaQKw}{Shanths-MBP-15.lan}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{ml.max_jvm_size=17179869184, ml.allocated_processors_double=12.0, xpack.installed=true, ml.machine_memory=34359738368, ml.allocated_processors=12}
[2022-12-20T21:59:24,858][INFO ][o.e.l.LicenseService     ] [Shanths-MBP-15.lan] license [e119d3da-b2fe-40ab-a32a-1d5565afd084] mode [basic] - valid
[2022-12-20T21:59:24,861][INFO ][o.e.g.GatewayService     ] [Shanths-MBP-15.lan] recovered [4] indices into cluster_state
[2022-12-20T21:59:24,957][INFO ][o.e.c.m.MetadataDeleteIndexService] [Shanths-MBP-15.lan] [.geoip_databases/H8rI72ZxSUWN7GUJJP187w] deleting index
[2022-12-20T21:59:24,965][INFO ][o.e.c.r.a.AllocationService] [Shanths-MBP-15.lan] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [deleted indices])." previous.health="YELLOW" reason="deleted indices"
[2022-12-20T21:59:25,196][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [Shanths-MBP-15.lan] Node [{Shanths-MBP-15.lan}{Gqqcnr5GRoKOol-P38vgrQ}] is selected as the current health node.
[2022-12-20T21:59:54,648][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] flood stage disk watermark [95%] exceeded on [Gqqcnr5GRoKOol-P38vgrQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 37.2gb[3.9%], all indices on this node will be marked read-only
[2022-12-20T22:00:24,652][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] flood stage disk watermark [95%] exceeded on [Gqqcnr5GRoKOol-P38vgrQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 37.2gb[3.9%], all indices on this node will be marked read-only

[2022-12-20T22:00:54,655][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] flood stage disk watermark [95%] exceeded on [Gqqcnr5GRoKOol-P38vgrQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 37.2gb[3.9%], all indices on this node will be marked read-only

Kibana error log

[2022-12-20T22:06:30.214+01:00][ERROR][savedobjects-service] [.kibana_task_manager] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_task_manager_8.5.3_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.
[2022-12-20T22:06:30.215+01:00][INFO ][savedobjects-service] [.kibana_task_manager] CREATE_NEW_TARGET -> CREATE_NEW_TARGET. took: 60027ms.
[2022-12-20T22:06:30.216+01:00][ERROR][savedobjects-service] [.kibana] Action failed with '[index_not_green_timeout] Timeout waiting for the status of the [.kibana_8.5.3_001] index to become 'green' Refer to https://www.elastic.co/guide/en/kibana/8.5/resolve-migrations-failures.html#_repeated_time_out_requests_that_eventually_fail for information on how to resolve the issue.'. Retrying attempt 1 in 2 seconds.

You're running out of disk space so Elasticsearch won't create more indices, you need to fix that.

I still have 135 GB left. Strange it complains about disk space...

Do you have another mount for Elasticsearch then?

I saw that 95% mark, cleaned up a bit now. It is mounted on /System/Volumes/Data (/dev/disk1s1), it shows in the log it has 83.8gb free. Does it need more storage than this? I am running this as a simple developer PC.

[2022-12-20T23:53:46,697][INFO ][o.e.e.NodeEnvironment    ] [Shanths-MBP-15.lan] using [1] data paths, mounts [[/System/Volumes/Data (/dev/disk1s1)]], net usable_space [83.8gb], net total_space [931.5gb], types [apfs]
[2022-12-20T23:53:46,698][INFO ][o.e.e.NodeEnvironment    ] [Shanths-MBP-15.lan] heap size [16gb], compressed ordinary object pointers [true]
[2022-12-20T23:53:46,805][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] node name [Shanths-MBP-15.lan], node ID [jbHQlMIhQ9a0dTjvxZMZZQ], cluster name [elasticsearch], roles [ingest, data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen]
[2022-12-20T23:53:50,537][INFO ][o.e.x.s.Security         ] [Shanths-MBP-15.lan] Security is disabled
[2022-12-20T23:53:50,624][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [Shanths-MBP-15.lan] [controller/81526] [Main.cc@123] controller (64 bit): Version 8.5.3 (Build 32c68d5e2f9f8f) Copyright (c) 2022 Elasticsearch BV
[2022-12-20T23:53:51,342][INFO ][o.e.t.n.NettyAllocator   ] [Shanths-MBP-15.lan] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2022-12-20T23:53:51,373][INFO ][o.e.i.r.RecoverySettings ] [Shanths-MBP-15.lan] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2022-12-20T23:53:51,426][INFO ][o.e.d.DiscoveryModule    ] [Shanths-MBP-15.lan] using discovery type [multi-node] and seed hosts providers [settings]
[2022-12-20T23:53:52,788][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] initialized
[2022-12-20T23:53:52,790][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] starting ...
[2022-12-20T23:53:52,871][INFO ][o.e.x.s.c.f.PersistentCache] [Shanths-MBP-15.lan] persistent cache index loaded
[2022-12-20T23:53:52,872][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [Shanths-MBP-15.lan] deprecation component started
[2022-12-20T23:53:53,030][INFO ][o.e.t.TransportService   ] [Shanths-MBP-15.lan] publish_address {192.168.10.132:9300}, bound_addresses {[::]:9300}
[2022-12-20T23:53:53,570][INFO ][o.e.b.BootstrapChecks    ] [Shanths-MBP-15.lan] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-12-20T23:53:53,572][WARN ][o.e.c.c.ClusterBootstrapService] [Shanths-MBP-15.lan] this node is locked into cluster UUID [9VWBkOTnRp2oMrlbfJwUbA] but [cluster.initial_master_nodes] is set to [Shanths-MBP-15.lan]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts
[2022-12-20T23:53:53,729][INFO ][o.e.c.s.MasterService    ] [Shanths-MBP-15.lan] elected-as-master ([1] nodes joined)[_FINISH_ELECTION_, {Shanths-MBP-15.lan}{jbHQlMIhQ9a0dTjvxZMZZQ}{PJQ43qYuSUWHRI0_BjM1_Q}{Shanths-MBP-15.lan}{192.168.10.132}{192.168.10.132:9300}{cdfhilmrstw} completing election], term: 5, version: 44, delta: master node changed {previous [], current [{Shanths-MBP-15.lan}{jbHQlMIhQ9a0dTjvxZMZZQ}{PJQ43qYuSUWHRI0_BjM1_Q}{Shanths-MBP-15.lan}{192.168.10.132}{192.168.10.132:9300}{cdfhilmrstw}]}
[2022-12-20T23:53:53,833][INFO ][o.e.c.s.ClusterApplierService] [Shanths-MBP-15.lan] master node changed {previous [], current [{Shanths-MBP-15.lan}{jbHQlMIhQ9a0dTjvxZMZZQ}{PJQ43qYuSUWHRI0_BjM1_Q}{Shanths-MBP-15.lan}{192.168.10.132}{192.168.10.132:9300}{cdfhilmrstw}]}, term: 5, version: 44, reason: Publication{term=5, version=44}
[2022-12-20T23:53:53,868][INFO ][o.e.r.s.FileSettingsService] [Shanths-MBP-15.lan] starting file settings watcher ...
[2022-12-20T23:53:53,876][INFO ][o.e.r.s.FileSettingsService] [Shanths-MBP-15.lan] file settings service up and running [tid=85]
[2022-12-20T23:53:53,882][INFO ][o.e.h.AbstractHttpServerTransport] [Shanths-MBP-15.lan] publish_address {192.168.10.132:9200}, bound_addresses {[::]:9200}
[2022-12-20T23:53:53,882][INFO ][o.e.n.Node               ] [Shanths-MBP-15.lan] started {Shanths-MBP-15.lan}{jbHQlMIhQ9a0dTjvxZMZZQ}{PJQ43qYuSUWHRI0_BjM1_Q}{Shanths-MBP-15.lan}{192.168.10.132}{192.168.10.132:9300}{cdfhilmrstw}{ml.allocated_processors=12, ml.machine_memory=34359738368, xpack.installed=true, ml.allocated_processors_double=12.0, ml.max_jvm_size=17179869184}
[2022-12-20T23:53:53,884][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] high disk watermark [90%] exceeded on [jbHQlMIhQ9a0dTjvxZMZZQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 83.8gb[9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2022-12-20T23:53:54,177][INFO ][o.e.l.LicenseService     ] [Shanths-MBP-15.lan] license [23050ceb-235d-4c31-a1cc-cae7f3e6244b] mode [basic] - valid
[2022-12-20T23:53:54,180][INFO ][o.e.g.GatewayService     ] [Shanths-MBP-15.lan] recovered [4] indices into cluster_state
[2022-12-20T23:53:54,288][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [Shanths-MBP-15.lan] Node [{Shanths-MBP-15.lan}{jbHQlMIhQ9a0dTjvxZMZZQ}] is selected as the current health node.
[2022-12-20T23:53:54,291][ERROR][o.e.i.g.GeoIpDownloader  ] [Shanths-MBP-15.lan] exception during geoip databases updateorg.elasticsearch.ElasticsearchException: not all primary shards of [.geoip_databases] index are active
	at org.elasticsearch.ingest.geoip@8.5.3/org.elasticsearch.ingest.geoip.GeoIpDownloader.updateDatabases(GeoIpDownloader.java:134)
	at org.elasticsearch.ingest.geoip@8.5.3/org.elasticsearch.ingest.geoip.GeoIpDownloader.runDownloader(GeoIpDownloader.java:274)
	at org.elasticsearch.ingest.geoip@8.5.3/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:102)
	at org.elasticsearch.ingest.geoip@8.5.3/org.elasticsearch.ingest.geoip.GeoIpDownloaderTaskExecutor.nodeOperation(GeoIpDownloaderTaskExecutor.java:48)
	at org.elasticsearch.server@8.5.3/org.elasticsearch.persistent.NodePersistentTasksExecutor$1.doRun(NodePersistentTasksExecutor.java:42)

See logs for more details.

[2022-12-20T23:53:54,371][INFO ][o.e.c.r.a.AllocationService] [Shanths-MBP-15.lan] current.health="RED" message="Cluster health status changed from [YELLOW] to [RED] (reason: [state recovered])." previous.health="YELLOW" reason="state recovered"
[2022-12-20T23:54:53,867][WARN ][o.e.c.r.a.DiskThresholdMonitor] [Shanths-MBP-15.lan] high disk watermark [90%] exceeded on [jbHQlMIhQ9a0dTjvxZMZZQ][Shanths-MBP-15.lan][/Users/shanthkumar/Work/Projects/ELK/elasticsearch-8.5.3/data] free: 83.8gb[9%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete

You might want to adjust the watermarks - Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block | Elasticsearch Guide [8.5] | Elastic

1 Like

Thanks for sharing the article, worked good!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.