Problem elastic. uneven distribution of the data node

Hi Team!
I'm using the elastic version 8.11.1.
Recently I began to notice an uneven distribution of indexes across data nodes.

Alert: Node v-elk-ed02 is reporting disk usage of 90% at February 2, 2024 3:44 PM EET

utilization

I mainly use 4 or 2 shards for indexes. Status cluster - green.
What is needed for data to be distributed evenly across all data nodes?

Thanks from your help.

You can set cluster.routing.allocation.balance.disk_usage to a higher value to make it a higher priority.

But, something to take note is that the shard movement will be more aggressive with the disk usage set as a higher priority.

Thanks,
how can I see the priority that is currently used for the cluster nodes? I also noticed that some shards with one index are located only on this node (all 4 are on one, but other shard nodes do not have this index).

This is an index without replicas, only 4 shards. And not only him.

The shard allocation heuristics setting priorities are mentioned in the doc I linked above.

You also can use the GET _cluster/settings?flat_settings&include_defaults API to get the all cluster settings including the default ones.

1 Like

Thanks from information.
Here is my command output, this is my first time encountering this and I don’t know what to do with it or where to start.
GET _cluster/settings?flat_settings&include_defaults

{
  "persistent": {
    "xpack.monitoring.collection.enabled": "true"
  },
  "transient": {},
  "defaults": {
    "action.auto_create_index": "true",
    "action.destructive_requires_name": "true",
    "action.search.pre_filter_shard_size.default": "128",
    "action.search.shard_count.limit": "9223372036854775807",
    "async_search.index_cleanup_interval": "1h",
    "bootstrap.ctrlhandler": "true",
    "bootstrap.memory_lock": "true",
    "cache.recycler.page.limit.heap": "10%",
    "cache.recycler.page.type": "CONCURRENT",
    "cache.recycler.page.weight.bytes": "1.0",
    "cache.recycler.page.weight.ints": "1.0",
    "cache.recycler.page.weight.longs": "1.0",
    "cache.recycler.page.weight.objects": "0.1",
    "ccr.auto_follow.wait_for_metadata_timeout": "60s",
    "ccr.indices.recovery.chunk_size": "1mb",
    "ccr.indices.recovery.internal_action_timeout": "60s",
    "ccr.indices.recovery.max_bytes_per_sec": "40mb",
    "ccr.indices.recovery.max_concurrent_file_chunks": "5",
    "ccr.indices.recovery.recovery_activity_timeout": "60s",
    "ccr.wait_for_metadata_timeout": "60s",
    "client.type": "node",
    "cluster.auto_shrink_voting_configuration": "true",
    "cluster.blocks.read_only": "false",
    "cluster.blocks.read_only_allow_delete": "false",
    "cluster.deprecation_indexing.enabled": "true",
    "cluster.deprecation_indexing.x_opaque_id_used.enabled": "true",
    "cluster.discovery_configuration_check.interval": "30000ms",
    "cluster.election.back_off_time": "100ms",
    "cluster.election.duration": "500ms",
    "cluster.election.initial_timeout": "100ms",
    "cluster.election.max_timeout": "10s",
    "cluster.election.strategy": "supports_voting_only",
    "cluster.fault_detection.follower_check.interval": "1000ms",
    "cluster.fault_detection.follower_check.retry_count": "3",
    "cluster.fault_detection.follower_check.timeout": "10000ms",
    "cluster.fault_detection.leader_check.interval": "1000ms",
    "cluster.fault_detection.leader_check.retry_count": "3",
    "cluster.fault_detection.leader_check.timeout": "10000ms",
    "cluster.follower_lag.timeout": "90000ms",
    "cluster.indices.close.enable": "true",
    "cluster.indices.tombstones.size": "500",
    "cluster.info.update.interval": "30s",
    "cluster.info.update.timeout": "15s",
    "cluster.initial_master_nodes": [],
    "cluster.join_validation.cache_timeout": "60s",
    "cluster.lifecycle.default.rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000",
    "cluster.max_shards_per_node": "1000",
    "cluster.max_shards_per_node.frozen": "3000",
    "cluster.max_voting_config_exclusions": "10",
    "cluster.name": "my-elk-prod",
    "cluster.no_master_block": "write",
    "cluster.nodes.reconnect_interval": "10s",
    "cluster.persistent_tasks.allocation.enable": "all",
    "cluster.persistent_tasks.allocation.recheck_interval": "30s",
    "cluster.publish.info_timeout": "10000ms",
    "cluster.publish.timeout": "30000ms",
    "cluster.remote.connections_per_cluster": "3",
    "cluster.remote.initial_connect_timeout": "30s",
    "cluster.remote.node.attr": "",
    "cluster.routing.allocation.allow_rebalance": "indices_all_active",
    "cluster.routing.allocation.awareness.attributes": [],
    "cluster.routing.allocation.balance.disk_usage": "2.0E-11",
    "cluster.routing.allocation.balance.index": "0.55",
    "cluster.routing.allocation.balance.shard": "0.45",
    "cluster.routing.allocation.balance.threshold": "1.0",
    "cluster.routing.allocation.balance.write_load": "10.0",
    "cluster.routing.allocation.cluster_concurrent_rebalance": "2",
    "cluster.routing.allocation.desired_balance.progress_log_interval": "1m",
    "cluster.routing.allocation.desired_balance.undesired_allocations.log_interval": "1h",
    "cluster.routing.allocation.desired_balance.undesired_allocations.threshold": "0.1",
    "cluster.routing.allocation.disk.reroute_interval": "60s",
    "cluster.routing.allocation.disk.threshold_enabled": "true",
    "cluster.routing.allocation.disk.watermark.enable_for_single_data_node": "true",
    "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen": "95%",
    "cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "20GB",
    "cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "100GB",
    "cluster.routing.allocation.disk.watermark.high": "90%",
    "cluster.routing.allocation.disk.watermark.high.max_headroom": "150GB",
    "cluster.routing.allocation.disk.watermark.low": "85%",
    "cluster.routing.allocation.disk.watermark.low.max_headroom": "200GB",
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.allocation.enforce_default_tier_preference": "true",
    "cluster.routing.allocation.node_concurrent_incoming_recoveries": "2",
    "cluster.routing.allocation.node_concurrent_outgoing_recoveries": "2",
    "cluster.routing.allocation.node_concurrent_recoveries": "2",
    "cluster.routing.allocation.node_initial_primaries_recoveries": "4",
    "cluster.routing.allocation.same_shard.host": "false",
    "cluster.routing.allocation.total_shards_per_node": "-1",
    "cluster.routing.allocation.type": "desired_balance",
    "cluster.routing.rebalance.enable": "all",
    "cluster.routing.use_adaptive_replica_selection": "true",
    "cluster.service.master_service_starvation_logging_threshold": "5m",
    "cluster.service.slow_master_task_logging_threshold": "10s",
    "cluster.service.slow_task_logging_threshold": "30s",
    "cluster.snapshot.info.max_concurrent_fetches": "5",
    "cluster_state.document_page_size": "1mb",
    "data_streams.lifecycle.poll_interval": "5m",
    "data_streams.lifecycle.target.merge.policy.floor_segment": "100mb",
    "data_streams.lifecycle.target.merge.policy.merge_factor": "16",
    "deprecation.skip_deprecated_settings": [],
    "discovery.cluster_formation_warning_timeout": "10000ms",
    "discovery.find_peers_interval": "1000ms",
    "discovery.initial_state_timeout": "30s",
    "discovery.probe.connect_timeout": "30s",
    "discovery.probe.handshake_timeout": "30s",
    "discovery.request_peers_timeout": "3000ms",
    "discovery.seed_hosts": [
      "v-elk-em01.my.cluster:9301",
      "v-elk-em02.my.cluster:9301",
      "v-elk-em03.my.cluster:9301"
    ],
    "discovery.seed_providers": [],
    "discovery.seed_resolver.max_concurrent_resolvers": "10",
    "discovery.seed_resolver.timeout": "5s",
    "discovery.type": "multi-node",
    "discovery.unconfigured_bootstrap_timeout": "3s",
    "enrich.cache_size": "1000",
    "enrich.cleanup_period": "15m",
    "enrich.coordinator_proxy.max_concurrent_requests": "8",
    "enrich.coordinator_proxy.max_lookups_per_request": "128",
    "enrich.coordinator_proxy.queue_capacity": "1024",
    "enrich.fetch_size": "10000",
    "enrich.max_concurrent_policy_executions": "50",
    "enrich.max_force_merge_attempts": "3",
    "esql.query.result_truncation_default_size": "500",
    "esql.query.result_truncation_max_size": "10000",
    "esql.queue_size": "1000",
    "esql.size": "4",
    "esql_worker.queue_size": "1000",
    "esql_worker.size": "7",
    "gateway.expected_data_nodes": "-1",
    "gateway.recover_after_data_nodes": "-1",
    "gateway.recover_after_time": "0ms",
    "gateway.slow_write_logging_threshold": "10s",
    "gateway.write_dangling_indices_info": "true",
    "health.ilm.max_retries_per_step": "100",
    "health.ilm.max_time_on_action": "1d",
    "health.ilm.max_time_on_step": "1d",
    "health.master_history.has_master_lookup_timeframe": "30s",
    "health.master_history.identity_changes_threshold": "4",
    "health.master_history.no_master_transitions_threshold": "4",
    "health.node.enabled": "true",
    "health.periodic_logger.enabled": "false",
    "health.periodic_logger.poll_interval": "60s",
    "health.reporting.local.monitor.interval": "30s",
    "health_node.transport_action_timeout": "5s",
    "http.bind_host": [
      "0.0.0.0"
    ],
    "http.client_stats.closed_channels.max_age": "5m",
    "http.client_stats.closed_channels.max_count": "10000",
    "http.client_stats.enabled": "true",
    "http.compression": "false",
    "http.compression_level": "3",
    "http.cors.allow-credentials": "false",
    "http.cors.allow-headers": "X-Requested-With,Content-Type,Content-Length,Authorization,Accept,User-Agent,X-Elastic-Client-Meta",
    "http.cors.allow-methods": "OPTIONS,HEAD,GET,POST,PUT,DELETE",
    "http.cors.allow-origin": "",
    "http.cors.enabled": "false",
    "http.cors.max-age": "1728000",
    "http.detailed_errors.enabled": "true",
    "http.host": [
      "0.0.0.0"
    ],
    "http.max_chunk_size": "8kb",
    "http.max_content_length": "100mb",
    "http.max_header_size": "16kb",
    "http.max_initial_line_length": "4kb",
    "http.max_warning_header_count": "-1",
    "http.max_warning_header_size": "-1b",
    "http.netty.max_composite_buffer_components": "69905",
    "http.netty.receive_predictor_size": "64kb",
    "http.netty.worker_count": "0",
    "http.pipelining.max_events": "10000",
    "http.port": "9200",
    "http.publish_host": [
      "0.0.0.0"
    ],
    "http.publish_port": "-1",
    "http.read_timeout": "0ms",
    "http.reset_cookies": "false",
    "http.shutdown_grace_period": "0ms",
    "http.tcp.keep_alive": "true",
    "http.tcp.keep_count": "-1",
    "http.tcp.keep_idle": "-1",
    "http.tcp.keep_interval": "-1",
    "http.tcp.no_delay": "true",
    "http.tcp.receive_buffer_size": "-1b",
    "http.tcp.reuse_address": "true",
    "http.tcp.send_buffer_size": "-1b",
    "http.tracer.exclude": [],
    "http.tracer.include": [],
    "http.type": "security4",
    "http.type.default": "netty4",
    "index.codec": "default",
    "index.recovery.type": "",
    "index.store.fs.fs_lock": "native",
    "index.store.preload": [],
    "index.store.snapshot.cache.excluded_file_types": [],
    "index.store.snapshot.uncached_chunk_size": "-1b",
    "index.store.type": "",
    "indexing_pressure.memory.limit": "10%",
    "indices.analysis.hunspell.dictionary.ignore_case": "false",
    "indices.analysis.hunspell.dictionary.lazy": "false",
    "indices.breaker.fielddata.limit": "40%",
    "indices.breaker.fielddata.overhead": "1.03",
    "indices.breaker.fielddata.type": "memory",
    "indices.breaker.request.limit": "60%",
    "indices.breaker.request.overhead": "1.0",
    "indices.breaker.request.type": "memory",
    "indices.breaker.total.limit": "95%",
    "indices.breaker.total.use_real_memory": "true",
    "indices.breaker.type": "hierarchy",
    "indices.cache.cleanup_interval": "1m",
    "indices.fielddata.cache.size": "-1b",
    "indices.id_field_data.enabled": "false",
    "indices.lifecycle.history_index_enabled": "true",
    "indices.lifecycle.poll_interval": "10m",
    "indices.lifecycle.rollover.only_if_has_documents": "true",
    "indices.lifecycle.step.master_timeout": "30s",
    "indices.mapping.dynamic_timeout": "30s",
    "indices.mapping.max_in_flight_updates": "10",
    "indices.memory.index_buffer_size": "10%",
    "indices.memory.interval": "5s",
    "indices.memory.max_index_buffer_size": "-1",
    "indices.memory.min_index_buffer_size": "48mb",
    "indices.memory.shard_inactive_time": "5m",
    "indices.queries.cache.all_segments": "false",
    "indices.queries.cache.count": "10000",
    "indices.queries.cache.size": "10%",
    "indices.query.bool.max_clause_count": "4096",
    "indices.query.bool.max_nested_depth": "30",
    "indices.query.query_string.allowLeadingWildcard": "true",
    "indices.query.query_string.analyze_wildcard": "false",
    "indices.recovery.internal_action_long_timeout": "1800000ms",
    "indices.recovery.internal_action_retry_timeout": "1m",
    "indices.recovery.internal_action_timeout": "15m",
    "indices.recovery.max_bytes_per_sec": "40mb",
    "indices.recovery.max_concurrent_file_chunks": "2",
    "indices.recovery.max_concurrent_operations": "1",
    "indices.recovery.max_concurrent_snapshot_file_downloads": "5",
    "indices.recovery.max_concurrent_snapshot_file_downloads_per_node": "25",
    "indices.recovery.recovery_activity_timeout": "1800000ms",
    "indices.recovery.retry_delay_network": "5s",
    "indices.recovery.retry_delay_state_sync": "500ms",
    "indices.recovery.use_snapshots": "true",
    "indices.replication.initial_retry_backoff_bound": "50ms",
    "indices.replication.retry_timeout": "60s",
    "indices.requests.cache.expire": "0ms",
    "indices.requests.cache.size": "1%",
    "indices.store.delete.shard.timeout": "30s",
    "indices.store.shard_lock_retry.interval": "1s",
    "indices.store.shard_lock_retry.timeout": "1m",
    "indices.write_ack_delay_interval": "0ms",
    "indices.write_ack_delay_randomness_bound": "70ms",
    "ingest.geoip.cache_size": "1000",
    "ingest.geoip.downloader.eager.download": "false",
    "ingest.geoip.downloader.enabled": "true",
    "ingest.geoip.downloader.endpoint": "https://geoip.elastic.co/v1/database",
    "ingest.geoip.downloader.poll.interval": "3d",
    "ingest.grok.watchdog.interval": "1s",
    "ingest.grok.watchdog.max_execution_time": "1s",
    "ingest.user_agent.cache_size": "1000",
    "ingest_node.transport_action_timeout": "20s",
    "logger.level": "INFO",
    "master_history.max_age": "30m",
    "monitor.fs.health.enabled": "true",
    "monitor.fs.health.refresh_interval": "120s",
    "monitor.fs.health.slow_path_logging_threshold": "5s",
    "monitor.fs.refresh_interval": "1s",
    "monitor.jvm.gc.enabled": "true",
    "monitor.jvm.gc.overhead.debug": "10",
    "monitor.jvm.gc.overhead.info": "25",
    "monitor.jvm.gc.overhead.warn": "50",
    "monitor.jvm.gc.refresh_interval": "1s",
    "monitor.jvm.refresh_interval": "1s",
    "monitor.os.refresh_interval": "1s",
    "monitor.process.refresh_interval": "1s",
    "network.bind_host": [
      "0"
    ],
    "network.breaker.inflight_requests.limit": "100%",
    "network.breaker.inflight_requests.overhead": "2.0",
    "network.host": [
      "0"
    ],
    "network.publish_host": [
      "0"
    ],
    "network.server": "true",
    "network.tcp.keep_alive": "true",
    "network.tcp.keep_count": "-1",
    "network.tcp.keep_idle": "-1",
    "network.tcp.keep_interval": "-1",
    "network.tcp.no_delay": "true",
    "network.tcp.receive_buffer_size": "-1b",
    "network.tcp.reuse_address": "true",
    "network.tcp.send_buffer_size": "-1b",
    "node._internal.default_refresh_interval": "1s",
    "node.attr.ml.config_version": "11.0.0",
    "node.attr.transform.config_version": "10.0.0",
    "node.attr.xpack.installed": "true",
    "node.bandwidth.recovery.disk.read": "-1",
    "node.bandwidth.recovery.disk.write": "-1",
    "node.bandwidth.recovery.factor.read": "0.4",
    "node.bandwidth.recovery.factor.write": "0.4",
    "node.bandwidth.recovery.network": "-1",
    "node.bandwidth.recovery.operator.factor": "0.4",
    "node.bandwidth.recovery.operator.factor.max_overcommit": "100.0",
    "node.bandwidth.recovery.operator.factor.read": "0.4",
    "node.bandwidth.recovery.operator.factor.write": "0.4",
    "node.enable_lucene_segment_infos_trace": "false",
    "node.external_id": "",
    "node.id.seed": "0",
    "node.name": "v-elk-ekb01.my.cluster",
    "node.portsfile": "false",
    "node.processors": "4.0",
    "node.roles": [],
    "node.store.allow_mmap": "true",
    "path.data": [
      "/var/lib/elasticsearch"
    ],
    "path.home": "/usr/share/elasticsearch",
    "path.logs": "/var/log/elasticsearch",
    "path.repo": [
      "/mnt/nfs-share"
    ],
    "path.shared_data": "",
    "plugin.mandatory": [],
    "readiness.port": "-1",
    "reindex.remote.whitelist": [],
    "remote_cluster.bind_host": [
      "0.0.0.0"
    ],
    "remote_cluster.host": [
      "0.0.0.0"
    ],
    "remote_cluster.max_request_header_size": "64kb",
    "remote_cluster.port": "9443",
    "remote_cluster.publish_host": [
      "0.0.0.0"
    ],
    "remote_cluster.publish_port": "-1",
    "remote_cluster.tcp.keep_alive": "true",
    "remote_cluster.tcp.keep_count": "-1",
    "remote_cluster.tcp.keep_idle": "-1",
    "remote_cluster.tcp.keep_interval": "-1",
    "remote_cluster.tcp.no_delay": "true",
    "remote_cluster.tcp.reuse_address": "true",
    "remote_cluster.tcp.send_buffer_size": "-1b",
    "remote_cluster_server.enabled": "false",
    "repositories.fs.chunk_size": "9223372036854775807b",
    "repositories.fs.location": "",
    "repositories.url.allowed_urls": [],
    "repositories.url.supported_protocols": [
      "http",
      "https",
      "ftp",
      "file",
      "jar"
    ],
    "repositories.url.url": "http:",
    "repository_s3.compare_and_exchange.anti_contention_delay": "1s",
    "repository_s3.compare_and_exchange.time_to_live": "15s",
    "resource.reload.enabled": "true",
    "resource.reload.interval.high": "5s",
    "resource.reload.interval.low": "60s",
    "resource.reload.interval.medium": "30s",
    "rest.action.multi.allow_explicit_index": "true",
    "runtime_fields.grok.watchdog.interval": "1s",
    "runtime_fields.grok.watchdog.max_execution_time": "1s",
    "script.allowed_contexts": [],
    "script.allowed_types": [],
    "script.cache.expire": "0ms",
    "script.cache.max_size": "3000",
    "script.disable_max_compilations_rate": "false",
    "script.max_compilations_rate": "150/5m",
    "script.max_size_in_bytes": "65535",
    "script.painless.regex.enabled": "limited",
    "script.painless.regex.limit-factor": "6",
    "search.aggs.rewrite_to_filter_by_filter": "true",
    "search.aggs.tdigest_execution_hint": "DEFAULT",
    "search.allow_expensive_queries": "true",
    "search.check_ccs_compatibility": "false",
    "search.default_allow_partial_results": "true",
    "search.default_keep_alive": "5m",
    "search.default_search_timeout": "-1",
    "search.highlight.term_vector_multi_value": "true",
    "search.keep_alive_interval": "1m",
    "search.low_level_cancellation": "true",
    "search.max_async_search_response_size": "10mb",
    "search.max_buckets": "65536",
    "search.max_keep_alive": "24h",
    "search.max_open_scroll_context": "500",
    "search.query_phase_parallel_collection_enabled": "false",
    "search.worker_threads_enabled": "true",
    "searchable_snapshots.blob_cache.periodic_cleanup.batch_size": "100",
    "searchable_snapshots.blob_cache.periodic_cleanup.interval": "1h",
    "searchable_snapshots.blob_cache.periodic_cleanup.pit_keep_alive": "10m",
    "searchable_snapshots.blob_cache.periodic_cleanup.retention_period": "1h",
    "security.manager.filter_bad_defaults": "true",
    "slm.health.failed_snapshot_warn_threshold": "5",
    "slm.history_index_enabled": "true",
    "slm.minimum_interval": "15m",
    "slm.retention_duration": "1h",
    "slm.retention_schedule": "0 30 1 * * ?",
    "snapshot.max_concurrent_operations": "1000",
    "snapshot.refresh_repo_uuid_on_restore": "true",
    "stack.templates.enabled": "true",
    "telemetry.metrics.enabled": "false",
    "thread_pool.analyze.queue_size": "16",
    "thread_pool.analyze.size": "1",
    "thread_pool.auto_complete.queue_size": "100",
    "thread_pool.auto_complete.size": "1",
    "thread_pool.azure_event_loop.core": "0",
    "thread_pool.azure_event_loop.keep_alive": "30s",
    "thread_pool.azure_event_loop.max": "1",
    "thread_pool.cluster_coordination.queue_size": "-1",
    "thread_pool.cluster_coordination.size": "1",
    "thread_pool.estimated_time_interval": "200ms",
    "thread_pool.estimated_time_interval.warn_threshold": "5s",
    "thread_pool.fetch_shard_started.core": "1",
    "thread_pool.fetch_shard_started.keep_alive": "5m",
    "thread_pool.fetch_shard_started.max": "8",
    "thread_pool.fetch_shard_store.core": "1",
    "thread_pool.fetch_shard_store.keep_alive": "5m",
    "thread_pool.fetch_shard_store.max": "8",
    "thread_pool.flush.core": "1",
    "thread_pool.flush.keep_alive": "5m",
    "thread_pool.flush.max": "2",
    "thread_pool.force_merge.queue_size": "-1",
    "thread_pool.force_merge.size": "1",
    "thread_pool.generic.core": "4",
    "thread_pool.generic.keep_alive": "30s",
    "thread_pool.generic.max": "128",
    "thread_pool.get.queue_size": "1000",
    "thread_pool.get.size": "7",
    "thread_pool.management.core": "1",
    "thread_pool.management.keep_alive": "5m",
    "thread_pool.management.max": "4",
    "thread_pool.profiling.core": "0",
    "thread_pool.profiling.keep_alive": "30m",
    "thread_pool.profiling.max": "1",
    "thread_pool.refresh.core": "1",
    "thread_pool.refresh.keep_alive": "5m",
    "thread_pool.refresh.max": "2",
    "thread_pool.repository_azure.core": "0",
    "thread_pool.repository_azure.keep_alive": "30s",
    "thread_pool.repository_azure.max": "5",
    "thread_pool.scheduler.warn_threshold": "5s",
    "thread_pool.search.queue_size": "1000",
    "thread_pool.search.size": "7",
    "thread_pool.search_coordination.queue_size": "1000",
    "thread_pool.search_coordination.size": "2",
    "thread_pool.search_throttled.queue_size": "100",
    "thread_pool.search_throttled.size": "1",
    "thread_pool.search_worker.queue_size": "-1",
    "thread_pool.search_worker.size": "7",
    "thread_pool.snapshot.core": "1",
    "thread_pool.snapshot.keep_alive": "5m",
    "thread_pool.snapshot.max": "10",
    "thread_pool.snapshot_meta.core": "1",
    "thread_pool.snapshot_meta.keep_alive": "30s",
    "thread_pool.snapshot_meta.max": "12",
    "thread_pool.system_critical_read.queue_size": "2000",
    "thread_pool.system_critical_read.size": "2",
    "thread_pool.system_critical_write.queue_size": "1500",
    "thread_pool.system_critical_write.size": "2",
    "thread_pool.system_read.queue_size": "2000",
    "thread_pool.system_read.size": "2",
    "thread_pool.system_write.queue_size": "1000",
    "thread_pool.system_write.size": "2",
    "thread_pool.warmer.core": "1",
    "thread_pool.warmer.keep_alive": "5m",
    "thread_pool.warmer.max": "2",
    "thread_pool.write.queue_size": "10000",
    "thread_pool.write.size": "4",
    "time_series.poll_interval": "5m",
    "tracing.apm.enabled": "false",
    "tracing.apm.names.exclude": [],
    "tracing.apm.names.include": [],
    "tracing.apm.sanitize_field_names": [
      "password",
      "passwd",
      "pwd",
      "secret",
      "*key",
      "*token*",
      "*session*",
      "*credit*",
      "*card*",
      "*auth*",
      "*principal*",
      "set-cookie"
    ],
    "transport.bind_host": [
      "0.0.0.0"
    ],
    "transport.compress": "INDEXING_DATA",
    "transport.compression_scheme": "LZ4",
    "transport.connect_timeout": "30s",
    "transport.connections_per_node.bulk": "3",
    "transport.connections_per_node.ping": "1",
    "transport.connections_per_node.recovery": "2",
    "transport.connections_per_node.reg": "6",
    "transport.connections_per_node.state": "1",
    "transport.host": [
      "0.0.0.0"
    ],
    "transport.netty.boss_count": "1",
    "transport.netty.receive_predictor_max": "64kb",
    "transport.netty.receive_predictor_min": "64kb",
    "transport.netty.receive_predictor_size": "64kb",
    "transport.netty.worker_count": "4",
    "transport.ping_schedule": "-1",
    "transport.port": "9301",
    "transport.publish_host": [
      "0.0.0.0"
    ],
    "transport.publish_port": "-1",
    "transport.rst_on_close": "false",
    "transport.slow_operation_logging_threshold": "5s",
    "transport.tcp.keep_alive": "true",
    "transport.tcp.keep_count": "-1",
    "transport.tcp.keep_idle": "-1",
    "transport.tcp.keep_interval": "-1",
    "transport.tcp.no_delay": "true",
    "transport.tcp.receive_buffer_size": "-1b",
    "transport.tcp.reuse_address": "true",
    "transport.tcp.send_buffer_size": "-1b",
    "transport.tracer.exclude": [
      "internal:coordination/fault_detection/*"
    ],
    "transport.tracer.include": [],
    "transport.type": "security4",
    "transport.type.default": "netty4",
    "write_load_forecaster.max_index_age": "7d",
    "xpack.applications.behavioral_analytics.ingest.bulk_processor.flush_delay": "10s",
    "xpack.applications.behavioral_analytics.ingest.bulk_processor.max_bytes_in_flight": "5%",
    "xpack.applications.behavioral_analytics.ingest.bulk_processor.max_events_per_bulk": "500",
    "xpack.applications.behavioral_analytics.ingest.bulk_processor.max_number_of_retries": "1",
    "xpack.applications.rules.max_rules_per_ruleset": "100",
    "xpack.autoscaling.memory.monitor.timeout": "15s",
    "xpack.ccr.ccr_thread_pool.queue_size": "100",
    "xpack.ccr.ccr_thread_pool.size": "32",
    "xpack.ccr.enabled": "true",
    "xpack.downsample.thread_pool.queue_size": "256",
    "xpack.downsample.thread_pool.size": "1",
    "xpack.ent_search.enabled": "true",
    "xpack.eql.enabled": "true",
    "xpack.graph.enabled": "true",
    "xpack.http.connection_pool_ttl": "-1",
    "xpack.http.default_connection_timeout": "10s",
    "xpack.http.default_read_timeout": "10s",
    "xpack.http.max_response_size": "10mb",
    "xpack.http.proxy.host": "",
    "xpack.http.proxy.port": "0",
    "xpack.http.proxy.scheme": "",
    "xpack.http.tcp.keep_alive": "true",
    "xpack.http.whitelist": [
      "*"
    ],
    "xpack.idp.allowed_nameid_formats": [
      "urn:oasis:names:tc:SAML:2.0:nameid-format:transient"
    ],
    "xpack.idp.contact.email": "",
    "xpack.idp.contact.given_name": "",
    "xpack.idp.contact.surname": "",
    "xpack.idp.defaults.authn_expiry": "5m",
    "xpack.idp.defaults.nameid_format": "urn:oasis:names:tc:SAML:2.0:nameid-format:transient",
    "xpack.idp.enabled": "false",
    "xpack.idp.entity_id": "",
    "xpack.idp.metadata.signing.keystore.alias": "",
    "xpack.idp.organization.display_name": "",
    "xpack.idp.organization.name": "",
    "xpack.idp.organization.url": "http:",
    "xpack.idp.privileges.application": "",
    "xpack.idp.privileges.cache.size": "100",
    "xpack.idp.privileges.cache.ttl": "90m",
    "xpack.idp.signing.keystore.alias": "",
    "xpack.idp.slo_endpoint.post": "https:",
    "xpack.idp.slo_endpoint.redirect": "https:",
    "xpack.idp.sp.cache.size": "1000",
    "xpack.idp.sp.cache.ttl": "60m",
    "xpack.idp.sp.wildcard.path": "wildcard_services.json",
    "xpack.idp.sso_endpoint.post": "https:",
    "xpack.idp.sso_endpoint.redirect": "https:",
    "xpack.inference.utility_thread_pool.core": "0",
    "xpack.inference.utility_thread_pool.keep_alive": "10m",
    "xpack.inference.utility_thread_pool.max": "1",
    "xpack.license.self_generated.type": "basic",
    "xpack.license.upload.types": [
      "standard",
      "gold",
      "platinum",
      "enterprise",
      "trial"
    ],
    "xpack.ml.allocated_processors_scale": "1",
    "xpack.ml.autodetect_process": "true",
    "xpack.ml.datafeed_thread_pool.core": "1",
    "xpack.ml.datafeed_thread_pool.keep_alive": "1m",
    "xpack.ml.datafeed_thread_pool.max": "512",
    "xpack.ml.delayed_data_check_freq": "15m",
    "xpack.ml.enable_config_migration": "true",
    "xpack.ml.enabled": "true",
    "xpack.ml.inference_model.cache_size": "40%",
    "xpack.ml.inference_model.time_to_live": "5m",
    "xpack.ml.job_comms_thread_pool.core": "4",
    "xpack.ml.job_comms_thread_pool.keep_alive": "1m",
    "xpack.ml.job_comms_thread_pool.max": "2048",
    "xpack.ml.max_anomaly_records": "500",
    "xpack.ml.max_inference_processors": "50",
    "xpack.ml.max_lazy_ml_nodes": "0",
    "xpack.ml.max_machine_memory_percent": "30",
    "xpack.ml.max_ml_node_size": "0b",
    "xpack.ml.max_model_memory_limit": "0b",
    "xpack.ml.max_open_jobs": "512",
    "xpack.ml.min_disk_space_off_heap": "5gb",
    "xpack.ml.model_repository": "https://ml-models.elastic.co",
    "xpack.ml.native_inference_comms_thread_pool.core": "3",
    "xpack.ml.native_inference_comms_thread_pool.keep_alive": "1m",
    "xpack.ml.native_inference_comms_thread_pool.max": "312",
    "xpack.ml.nightly_maintenance_requests_per_second": "-1.0",
    "xpack.ml.node_concurrent_job_allocations": "2",
    "xpack.ml.persist_results_max_retries": "20",
    "xpack.ml.process_connect_timeout": "10s",
    "xpack.ml.use_auto_machine_memory_percent": "false",
    "xpack.ml.utility_thread_pool.core": "1",
    "xpack.ml.utility_thread_pool.keep_alive": "10m",
    "xpack.ml.utility_thread_pool.max": "2048",
    "xpack.monitoring.collection.ccr.stats.timeout": "10s",
    "xpack.monitoring.collection.cluster.stats.timeout": "10s",
    "xpack.monitoring.collection.enrich.stats.timeout": "10s",
    "xpack.monitoring.collection.index.recovery.active_only": "false",
    "xpack.monitoring.collection.index.recovery.timeout": "10s",
    "xpack.monitoring.collection.index.stats.timeout": "10s",
    "xpack.monitoring.collection.indices": [],
    "xpack.monitoring.collection.interval": "10s",
    "xpack.monitoring.collection.ml.job.stats.timeout": "10s",
    "xpack.monitoring.collection.node.stats.timeout": "10s",
    "xpack.monitoring.elasticsearch.collection.enabled": "true",
    "xpack.monitoring.history.duration": "168h",
    "xpack.monitoring.migration.decommission_alerts": "false",
    "xpack.monitoring.templates.enabled": "true",
    "xpack.notification.email.account.domain_allowlist": [
      "*"
    ],
    "xpack.notification.email.default_account": "",
    "xpack.notification.email.html.sanitization.allow": [
      "body",
      "head",
      "_tables",
      "_links",
      "_blocks",
      "_formatting",
      "img:embedded"
    ],
    "xpack.notification.email.html.sanitization.disallow": [],
    "xpack.notification.email.html.sanitization.enabled": "true",
    "xpack.notification.jira.default_account": "",
    "xpack.notification.pagerduty.default_account": "",
    "xpack.notification.reporting.interval": "15s",
    "xpack.notification.reporting.retries": "40",
    "xpack.notification.reporting.warning.enabled": "true",
    "xpack.notification.slack.default_account": "",
    "xpack.notification.webhook.additional_token_enabled": "false",
    "xpack.profiling.check_outdated_indices": "true",
    "xpack.profiling.enabled": "true",
    "xpack.profiling.query.details.max_slices": "16",
    "xpack.profiling.query.realtime": "true",
    "xpack.profiling.query.stacktrace.max_slices": "16",
    "xpack.profiling.templates.enabled": "false",
    "xpack.rollup.task_thread_pool.queue_size": "-1",
    "xpack.rollup.task_thread_pool.size": "1",
    "xpack.searchable.snapshot.allocate_on_rolling_restart": "false",
    "xpack.searchable.snapshot.cache.range_size": "32mb",
    "xpack.searchable.snapshot.cache.recovery_range_size": "128kb",
    "xpack.searchable.snapshot.cache.sync.interval": "60s",
    "xpack.searchable.snapshot.cache.sync.max_files": "10000",
    "xpack.searchable.snapshot.cache.sync.shutdown_timeout": "10s",
    "xpack.searchable.snapshot.shared_cache.count_reads": "true",
    "xpack.searchable.snapshot.shared_cache.decay.interval": "60s",
    "xpack.searchable.snapshot.shared_cache.max_freq": "100",
    "xpack.searchable.snapshot.shared_cache.min_time_delta": "60s",
    "xpack.searchable.snapshot.shared_cache.mmap": "false",
    "xpack.searchable.snapshot.shared_cache.range_size": "16mb",
    "xpack.searchable.snapshot.shared_cache.recovery_range_size": "128kb",
    "xpack.searchable.snapshot.shared_cache.region_size": "16mb",
    "xpack.searchable.snapshot.shared_cache.size": "0",
    "xpack.searchable.snapshot.shared_cache.size.max_headroom": "-1",
    "xpack.searchable_snapshots.cache_fetch_async_thread_pool.core": "0",
    "xpack.searchable_snapshots.cache_fetch_async_thread_pool.keep_alive": "30s",
    "xpack.searchable_snapshots.cache_fetch_async_thread_pool.max": "12",
    "xpack.searchable_snapshots.cache_prewarming_thread_pool.core": "0",
    "xpack.searchable_snapshots.cache_prewarming_thread_pool.keep_alive": "30s",
    "xpack.searchable_snapshots.cache_prewarming_thread_pool.max": "16",
    "xpack.security.audit.enabled": "false",
    "xpack.security.audit.logfile.emit_cluster_name": "false",
    "xpack.security.audit.logfile.emit_cluster_uuid": "true",
    "xpack.security.audit.logfile.emit_node_host_address": "false",
    "xpack.security.audit.logfile.emit_node_host_name": "false",
    "xpack.security.audit.logfile.emit_node_id": "true",
    "xpack.security.audit.logfile.emit_node_name": "false",
    "xpack.security.audit.logfile.events.emit_request_body": "false",
    "xpack.security.audit.logfile.events.exclude": [],
    "xpack.security.audit.logfile.events.include": [
      "ACCESS_DENIED",
      "ACCESS_GRANTED",
      "ANONYMOUS_ACCESS_DENIED",
      "AUTHENTICATION_FAILED",
      "CONNECTION_DENIED",
      "TAMPERED_REQUEST",
      "RUN_AS_DENIED",
      "RUN_AS_GRANTED",
      "SECURITY_CONFIG_CHANGE"
    ],
    "xpack.security.authc.anonymous.authz_exception": "true",
    "xpack.security.authc.anonymous.roles": [],
    "xpack.security.authc.anonymous.username": "_anonymous",
    "xpack.security.authc.api_key.cache.hash_algo": "ssha256",
    "xpack.security.authc.api_key.cache.max_keys": "25000",
    "xpack.security.authc.api_key.cache.ttl": "24h",
    "xpack.security.authc.api_key.delete.interval": "24h",
    "xpack.security.authc.api_key.delete.retention_period": "7d",
    "xpack.security.authc.api_key.delete.timeout": "-1",
    "xpack.security.authc.api_key.doc_cache.ttl": "5m",
    "xpack.security.authc.api_key.enabled": "true",
    "xpack.security.authc.api_key.hashing.algorithm": "PBKDF2",
    "xpack.security.authc.password_hashing.algorithm": "BCRYPT",
    "xpack.security.authc.reserved_realm.enabled": "true",
    "xpack.security.authc.run_as.enabled": "true",
    "xpack.security.authc.service_token.cache.hash_algo": "ssha256",
    "xpack.security.authc.service_token.cache.max_tokens": "100000",
    "xpack.security.authc.service_token.cache.ttl": "20m",
    "xpack.security.authc.success_cache.enabled": "true",
    "xpack.security.authc.success_cache.expire_after_access": "1h",
    "xpack.security.authc.success_cache.size": "10000",
    "xpack.security.authc.token.delete.interval": "30m",
    "xpack.security.authc.token.delete.timeout": "-1",
    "xpack.security.authc.token.enabled": "false",
    "xpack.security.authc.token.thread_pool.queue_size": "1000",
    "xpack.security.authc.token.thread_pool.size": "1",
    "xpack.security.authc.token.timeout": "20m",
    "xpack.security.authz.store.privileges.cache.max_size": "10000",
    "xpack.security.authz.store.privileges.cache.ttl": "24h",
    "xpack.security.authz.store.roles.cache.max_size": "10000",
    "xpack.security.authz.store.roles.field_permissions.cache.max_size_in_bytes": "104857600",
    "xpack.security.authz.store.roles.has_privileges.cache.max_size": "1000",
    "xpack.security.authz.store.roles.negative_lookup_cache.max_size": "10000",
    "xpack.security.authz.timer.indices.enabled": "false",
    "xpack.security.authz.timer.indices.threshold.debug": "20ms",
    "xpack.security.authz.timer.indices.threshold.info": "100ms",
    "xpack.security.authz.timer.indices.threshold.warn": "200ms",
    "xpack.security.autoconfiguration.enabled": "true",
    "xpack.security.automata.cache.enabled": "true",
    "xpack.security.automata.cache.size": "10000",
    "xpack.security.automata.cache.ttl": "48h",
    "xpack.security.automata.max_determinized_states": "100000",
    "xpack.security.crypto.thread_pool.queue_size": "1000",
    "xpack.security.crypto.thread_pool.size": "2",
    "xpack.security.dls.bitset.cache.size": "10%",
    "xpack.security.dls.bitset.cache.ttl": "2h",
    "xpack.security.dls_fls.enabled": "true",
    "xpack.security.enabled": "true",
    "xpack.security.encryption.algorithm": "AES/CTR/NoPadding",
    "xpack.security.encryption_key.algorithm": "AES",
    "xpack.security.encryption_key.length": "128",
    "xpack.security.enrollment.enabled": "false",
    "xpack.security.filter.always_allow_bound_address": "true",
    "xpack.security.fips_mode.enabled": "false",
    "xpack.security.http.filter.allow": [],
    "xpack.security.http.filter.deny": [],
    "xpack.security.http.filter.enabled": "true",
    "xpack.security.http.ssl.enabled": "true",
    "xpack.security.operator_privileges.enabled": "false",
    "xpack.security.remote_cluster.filter.allow": [],
    "xpack.security.remote_cluster.filter.deny": [],
    "xpack.security.remote_cluster_client.ssl.enabled": "true",
    "xpack.security.remote_cluster_server.ssl.enabled": "true",
    "xpack.security.ssl.diagnose.trust": "true",
    "xpack.security.transport.filter.allow": [],
    "xpack.security.transport.filter.deny": [],
    "xpack.security.transport.filter.enabled": "true",
    "xpack.security.transport.ssl.enabled": "true",
    "xpack.security.user": null,
    "xpack.transform.num_transform_failure_retries": "10",
    "xpack.transform.transform_scheduler_frequency": "1s",
    "xpack.watcher.actions.bulk.default_timeout": "",
    "xpack.watcher.actions.index.default_timeout": "",
    "xpack.watcher.bulk.actions": "1",
    "xpack.watcher.bulk.concurrent_requests": "0",
    "xpack.watcher.bulk.flush_interval": "1s",
    "xpack.watcher.bulk.size": "1mb",
    "xpack.watcher.enabled": "true",
    "xpack.watcher.encrypt_sensitive_data": "false",
    "xpack.watcher.execution.default_throttle_period": "5s",
    "xpack.watcher.execution.scroll.size": "0",
    "xpack.watcher.execution.scroll.timeout": "",
    "xpack.watcher.index.rest.direct_access": "",
    "xpack.watcher.input.search.default_timeout": "",
    "xpack.watcher.internal.ops.bulk.default_timeout": "",
    "xpack.watcher.internal.ops.index.default_timeout": "",
    "xpack.watcher.internal.ops.search.default_timeout": "",
    "xpack.watcher.stop.timeout": "30s",
    "xpack.watcher.thread_pool.queue_size": "1000",
    "xpack.watcher.thread_pool.size": "1",
    "xpack.watcher.transform.search.default_timeout": "",
    "xpack.watcher.trigger.schedule.ticker.tick_interval": "500ms",
    "xpack.watcher.use_ilm_index_management": "true",
    "xpack.watcher.watch.scroll.size": "0"
  }
}
}

I currently have this value set to:

 "cluster.routing.allocation.balance.disk_usage": "2.0E-11",

What value would you recommend setting?

You can use the PUT _cluster/settings API to set that setting value to 1.0.

1 Like

Do I understand correctly that you need to execute the request:

PUT _cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.balance.disk_usage": 1.0
  }
}

???

now GET _cluster/settings

{
  "persistent": {
    "xpack": {
      "monitoring": {
        "collection": {
          "enabled": "true"
        }
      }
    }
  },
  "transient": {}
}

thanks

Yup, you should start seeing shards being moved around.

thanks, Now the situation has improved a little. It looks like this:

I understand that the situation will not improve? from 150 to 279 GB