Hi Team!
I'm using the elastic version 8.11.1.
Recently I began to notice an uneven distribution of indexes across data nodes.
Alert: Node v-elk-ed02 is reporting disk usage of 90% at February 2, 2024 3:44 PM EET
utilization
I mainly use 4 or 2 shards for indexes. Status cluster - green.
What is needed for data to be distributed evenly across all data nodes?
Thanks from your help.
You can set cluster.routing.allocation.balance.disk_usage to a higher value to make it a higher priority.
But, something to take note is that the shard movement will be more aggressive with the disk usage set as a higher priority.
Thanks,
how can I see the priority that is currently used for the cluster nodes? I also noticed that some shards with one index are located only on this node (all 4 are on one, but other shard nodes do not have this index).
This is an index without replicas, only 4 shards. And not only him.
The shard allocation heuristics setting priorities are mentioned in the doc I linked above.
You also can use the GET _cluster/settings?flat_settings&include_defaults API to get the all cluster settings including the default ones.
Thanks from information.
Here is my command output, this is my first time encountering this and I don’t know what to do with it or where to start.
GET _cluster/settings?flat_settings&include_defaults
{
"persistent": {
"xpack.monitoring.collection.enabled": "true"
},
"transient": {},
"defaults": {
"action.auto_create_index": "true",
"action.destructive_requires_name": "true",
"action.search.pre_filter_shard_size.default": "128",
"action.search.shard_count.limit": "9223372036854775807",
"async_search.index_cleanup_interval": "1h",
"bootstrap.ctrlhandler": "true",
"bootstrap.memory_lock": "true",
"cache.recycler.page.limit.heap": "10%",
"cache.recycler.page.type": "CONCURRENT",
"cache.recycler.page.weight.bytes": "1.0",
"cache.recycler.page.weight.ints": "1.0",
"cache.recycler.page.weight.longs": "1.0",
"cache.recycler.page.weight.objects": "0.1",
"ccr.auto_follow.wait_for_metadata_timeout": "60s",
"ccr.indices.recovery.chunk_size": "1mb",
"ccr.indices.recovery.internal_action_timeout": "60s",
"ccr.indices.recovery.max_bytes_per_sec": "40mb",
"ccr.indices.recovery.max_concurrent_file_chunks": "5",
"ccr.indices.recovery.recovery_activity_timeout": "60s",
"ccr.wait_for_metadata_timeout": "60s",
"client.type": "node",
"cluster.auto_shrink_voting_configuration": "true",
"cluster.blocks.read_only": "false",
"cluster.blocks.read_only_allow_delete": "false",
"cluster.deprecation_indexing.enabled": "true",
"cluster.deprecation_indexing.x_opaque_id_used.enabled": "true",
"cluster.discovery_configuration_check.interval": "30000ms",
"cluster.election.back_off_time": "100ms",
"cluster.election.duration": "500ms",
"cluster.election.initial_timeout": "100ms",
"cluster.election.max_timeout": "10s",
"cluster.election.strategy": "supports_voting_only",
"cluster.fault_detection.follower_check.interval": "1000ms",
"cluster.fault_detection.follower_check.retry_count": "3",
"cluster.fault_detection.follower_check.timeout": "10000ms",
"cluster.fault_detection.leader_check.interval": "1000ms",
"cluster.fault_detection.leader_check.retry_count": "3",
"cluster.fault_detection.leader_check.timeout": "10000ms",
"cluster.follower_lag.timeout": "90000ms",
"cluster.indices.close.enable": "true",
"cluster.indices.tombstones.size": "500",
"cluster.info.update.interval": "30s",
"cluster.info.update.timeout": "15s",
"cluster.initial_master_nodes": [],
"cluster.join_validation.cache_timeout": "60s",
"cluster.lifecycle.default.rollover": "max_age=auto,max_primary_shard_size=50gb,min_docs=1,max_primary_shard_docs=200000000",
"cluster.max_shards_per_node": "1000",
"cluster.max_shards_per_node.frozen": "3000",
"cluster.max_voting_config_exclusions": "10",
"cluster.name": "my-elk-prod",
"cluster.no_master_block": "write",
"cluster.nodes.reconnect_interval": "10s",
"cluster.persistent_tasks.allocation.enable": "all",
"cluster.persistent_tasks.allocation.recheck_interval": "30s",
"cluster.publish.info_timeout": "10000ms",
"cluster.publish.timeout": "30000ms",
"cluster.remote.connections_per_cluster": "3",
"cluster.remote.initial_connect_timeout": "30s",
"cluster.remote.node.attr": "",
"cluster.routing.allocation.allow_rebalance": "indices_all_active",
"cluster.routing.allocation.awareness.attributes": [],
"cluster.routing.allocation.balance.disk_usage": "2.0E-11",
"cluster.routing.allocation.balance.index": "0.55",
"cluster.routing.allocation.balance.shard": "0.45",
"cluster.routing.allocation.balance.threshold": "1.0",
"cluster.routing.allocation.balance.write_load": "10.0",
"cluster.routing.allocation.cluster_concurrent_rebalance": "2",
"cluster.routing.allocation.desired_balance.progress_log_interval": "1m",
"cluster.routing.allocation.desired_balance.undesired_allocations.log_interval": "1h",
"cluster.routing.allocation.desired_balance.undesired_allocations.threshold": "0.1",
"cluster.routing.allocation.disk.reroute_interval": "60s",
"cluster.routing.allocation.disk.threshold_enabled": "true",
"cluster.routing.allocation.disk.watermark.enable_for_single_data_node": "true",
"cluster.routing.allocation.disk.watermark.flood_stage": "95%",
"cluster.routing.allocation.disk.watermark.flood_stage.frozen": "95%",
"cluster.routing.allocation.disk.watermark.flood_stage.frozen.max_headroom": "20GB",
"cluster.routing.allocation.disk.watermark.flood_stage.max_headroom": "100GB",
"cluster.routing.allocation.disk.watermark.high": "90%",
"cluster.routing.allocation.disk.watermark.high.max_headroom": "150GB",
"cluster.routing.allocation.disk.watermark.low": "85%",
"cluster.routing.allocation.disk.watermark.low.max_headroom": "200GB",
"cluster.routing.allocation.enable": "all",
"cluster.routing.allocation.enforce_default_tier_preference": "true",
"cluster.routing.allocation.node_concurrent_incoming_recoveries": "2",
"cluster.routing.allocation.node_concurrent_outgoing_recoveries": "2",
"cluster.routing.allocation.node_concurrent_recoveries": "2",
"cluster.routing.allocation.node_initial_primaries_recoveries": "4",
"cluster.routing.allocation.same_shard.host": "false",
"cluster.routing.allocation.total_shards_per_node": "-1",
"cluster.routing.allocation.type": "desired_balance",
"cluster.routing.rebalance.enable": "all",
"cluster.routing.use_adaptive_replica_selection": "true",
"cluster.service.master_service_starvation_logging_threshold": "5m",
"cluster.service.slow_master_task_logging_threshold": "10s",
"cluster.service.slow_task_logging_threshold": "30s",
"cluster.snapshot.info.max_concurrent_fetches": "5",
"cluster_state.document_page_size": "1mb",
"data_streams.lifecycle.poll_interval": "5m",
"data_streams.lifecycle.target.merge.policy.floor_segment": "100mb",
"data_streams.lifecycle.target.merge.policy.merge_factor": "16",
"deprecation.skip_deprecated_settings": [],
"discovery.cluster_formation_warning_timeout": "10000ms",
"discovery.find_peers_interval": "1000ms",
"discovery.initial_state_timeout": "30s",
"discovery.probe.connect_timeout": "30s",
"discovery.probe.handshake_timeout": "30s",
"discovery.request_peers_timeout": "3000ms",
"discovery.seed_hosts": [
"v-elk-em01.my.cluster:9301",
"v-elk-em02.my.cluster:9301",
"v-elk-em03.my.cluster:9301"
],
"discovery.seed_providers": [],
"discovery.seed_resolver.max_concurrent_resolvers": "10",
"discovery.seed_resolver.timeout": "5s",
"discovery.type": "multi-node",
"discovery.unconfigured_bootstrap_timeout": "3s",
"enrich.cache_size": "1000",
"enrich.cleanup_period": "15m",
"enrich.coordinator_proxy.max_concurrent_requests": "8",
"enrich.coordinator_proxy.max_lookups_per_request": "128",
"enrich.coordinator_proxy.queue_capacity": "1024",
"enrich.fetch_size": "10000",
"enrich.max_concurrent_policy_executions": "50",
"enrich.max_force_merge_attempts": "3",
"esql.query.result_truncation_default_size": "500",
"esql.query.result_truncation_max_size": "10000",
"esql.queue_size": "1000",
"esql.size": "4",
"esql_worker.queue_size": "1000",
"esql_worker.size": "7",
"gateway.expected_data_nodes": "-1",
"gateway.recover_after_data_nodes": "-1",
"gateway.recover_after_time": "0ms",
"gateway.slow_write_logging_threshold": "10s",
"gateway.write_dangling_indices_info": "true",
"health.ilm.max_retries_per_step": "100",
"health.ilm.max_time_on_action": "1d",
"health.ilm.max_time_on_step": "1d",
"health.master_history.has_master_lookup_timeframe": "30s",
"health.master_history.identity_changes_threshold": "4",
"health.master_history.no_master_transitions_threshold": "4",
"health.node.enabled": "true",
"health.periodic_logger.enabled": "false",
"health.periodic_logger.poll_interval": "60s",
"health.reporting.local.monitor.interval": "30s",
"health_node.transport_action_timeout": "5s",
"http.bind_host": [
"0.0.0.0"
],
"http.client_stats.closed_channels.max_age": "5m",
"http.client_stats.closed_channels.max_count": "10000",
"http.client_stats.enabled": "true",
"http.compression": "false",
"http.compression_level": "3",
"http.cors.allow-credentials": "false",
"http.cors.allow-headers": "X-Requested-With,Content-Type,Content-Length,Authorization,Accept,User-Agent,X-Elastic-Client-Meta",
"http.cors.allow-methods": "OPTIONS,HEAD,GET,POST,PUT,DELETE",
"http.cors.allow-origin": "",
"http.cors.enabled": "false",
"http.cors.max-age": "1728000",
"http.detailed_errors.enabled": "true",
"http.host": [
"0.0.0.0"
],
"http.max_chunk_size": "8kb",
"http.max_content_length": "100mb",
"http.max_header_size": "16kb",
"http.max_initial_line_length": "4kb",
"http.max_warning_header_count": "-1",
"http.max_warning_header_size": "-1b",
"http.netty.max_composite_buffer_components": "69905",
"http.netty.receive_predictor_size": "64kb",
"http.netty.worker_count": "0",
"http.pipelining.max_events": "10000",
"http.port": "9200",
"http.publish_host": [
"0.0.0.0"
],
"http.publish_port": "-1",
"http.read_timeout": "0ms",
"http.reset_cookies": "false",
"http.shutdown_grace_period": "0ms",
"http.tcp.keep_alive": "true",
"http.tcp.keep_count": "-1",
"http.tcp.keep_idle": "-1",
"http.tcp.keep_interval": "-1",
"http.tcp.no_delay": "true",
"http.tcp.receive_buffer_size": "-1b",
"http.tcp.reuse_address": "true",
"http.tcp.send_buffer_size": "-1b",
"http.tracer.exclude": [],
"http.tracer.include": [],
"http.type": "security4",
"http.type.default": "netty4",
"index.codec": "default",
"index.recovery.type": "",
"index.store.fs.fs_lock": "native",
"index.store.preload": [],
"index.store.snapshot.cache.excluded_file_types": [],
"index.store.snapshot.uncached_chunk_size": "-1b",
"index.store.type": "",
"indexing_pressure.memory.limit": "10%",
"indices.analysis.hunspell.dictionary.ignore_case": "false",
"indices.analysis.hunspell.dictionary.lazy": "false",
"indices.breaker.fielddata.limit": "40%",
"indices.breaker.fielddata.overhead": "1.03",
"indices.breaker.fielddata.type": "memory",
"indices.breaker.request.limit": "60%",
"indices.breaker.request.overhead": "1.0",
"indices.breaker.request.type": "memory",
"indices.breaker.total.limit": "95%",
"indices.breaker.total.use_real_memory": "true",
"indices.breaker.type": "hierarchy",
"indices.cache.cleanup_interval": "1m",
"indices.fielddata.cache.size": "-1b",
"indices.id_field_data.enabled": "false",
"indices.lifecycle.history_index_enabled": "true",
"indices.lifecycle.poll_interval": "10m",
"indices.lifecycle.rollover.only_if_has_documents": "true",
"indices.lifecycle.step.master_timeout": "30s",
"indices.mapping.dynamic_timeout": "30s",
"indices.mapping.max_in_flight_updates": "10",
"indices.memory.index_buffer_size": "10%",
"indices.memory.interval": "5s",
"indices.memory.max_index_buffer_size": "-1",
"indices.memory.min_index_buffer_size": "48mb",
"indices.memory.shard_inactive_time": "5m",
"indices.queries.cache.all_segments": "false",
"indices.queries.cache.count": "10000",
"indices.queries.cache.size": "10%",
"indices.query.bool.max_clause_count": "4096",
"indices.query.bool.max_nested_depth": "30",
"indices.query.query_string.allowLeadingWildcard": "true",
"indices.query.query_string.analyze_wildcard": "false",
"indices.recovery.internal_action_long_timeout": "1800000ms",
"indices.recovery.internal_action_retry_timeout": "1m",
"indices.recovery.internal_action_timeout": "15m",
"indices.recovery.max_bytes_per_sec": "40mb",
"indices.recovery.max_concurrent_file_chunks": "2",
"indices.recovery.max_concurrent_operations": "1",
"indices.recovery.max_concurrent_snapshot_file_downloads": "5",
"indices.recovery.max_concurrent_snapshot_file_downloads_per_node": "25",
"indices.recovery.recovery_activity_timeout": "1800000ms",
"indices.recovery.retry_delay_network": "5s",
"indices.recovery.retry_delay_state_sync": "500ms",
"indices.recovery.use_snapshots": "true",
"indices.replication.initial_retry_backoff_bound": "50ms",
"indices.replication.retry_timeout": "60s",
"indices.requests.cache.expire": "0ms",
"indices.requests.cache.size": "1%",
"indices.store.delete.shard.timeout": "30s",
"indices.store.shard_lock_retry.interval": "1s",
"indices.store.shard_lock_retry.timeout": "1m",
"indices.write_ack_delay_interval": "0ms",
"indices.write_ack_delay_randomness_bound": "70ms",
"ingest.geoip.cache_size": "1000",
"ingest.geoip.downloader.eager.download": "false",
"ingest.geoip.downloader.enabled": "true",
"ingest.geoip.downloader.endpoint": "https://geoip.elastic.co/v1/database",
"ingest.geoip.downloader.poll.interval": "3d",
"ingest.grok.watchdog.interval": "1s",
"ingest.grok.watchdog.max_execution_time": "1s",
"ingest.user_agent.cache_size": "1000",
"ingest_node.transport_action_timeout": "20s",
"logger.level": "INFO",
"master_history.max_age": "30m",
"monitor.fs.health.enabled": "true",
"monitor.fs.health.refresh_interval": "120s",
"monitor.fs.health.slow_path_logging_threshold": "5s",
"monitor.fs.refresh_interval": "1s",
"monitor.jvm.gc.enabled": "true",
"monitor.jvm.gc.overhead.debug": "10",
"monitor.jvm.gc.overhead.info": "25",
"monitor.jvm.gc.overhead.warn": "50",
"monitor.jvm.gc.refresh_interval": "1s",
"monitor.jvm.refresh_interval": "1s",
"monitor.os.refresh_interval": "1s",
"monitor.process.refresh_interval": "1s",
"network.bind_host": [
"0"
],
"network.breaker.inflight_requests.limit": "100%",
"network.breaker.inflight_requests.overhead": "2.0",
"network.host": [
"0"
],
"network.publish_host": [
"0"
],
"network.server": "true",
"network.tcp.keep_alive": "true",
"network.tcp.keep_count": "-1",
"network.tcp.keep_idle": "-1",
"network.tcp.keep_interval": "-1",
"network.tcp.no_delay": "true",
"network.tcp.receive_buffer_size": "-1b",
"network.tcp.reuse_address": "true",
"network.tcp.send_buffer_size": "-1b",
"node._internal.default_refresh_interval": "1s",
"node.attr.ml.config_version": "11.0.0",
"node.attr.transform.config_version": "10.0.0",
"node.attr.xpack.installed": "true",
"node.bandwidth.recovery.disk.read": "-1",
"node.bandwidth.recovery.disk.write": "-1",
"node.bandwidth.recovery.factor.read": "0.4",
"node.bandwidth.recovery.factor.write": "0.4",
"node.bandwidth.recovery.network": "-1",
"node.bandwidth.recovery.operator.factor": "0.4",
"node.bandwidth.recovery.operator.factor.max_overcommit": "100.0",
"node.bandwidth.recovery.operator.factor.read": "0.4",
"node.bandwidth.recovery.operator.factor.write": "0.4",
"node.enable_lucene_segment_infos_trace": "false",
"node.external_id": "",
"node.id.seed": "0",
"node.name": "v-elk-ekb01.my.cluster",
"node.portsfile": "false",
"node.processors": "4.0",
"node.roles": [],
"node.store.allow_mmap": "true",
"path.data": [
"/var/lib/elasticsearch"
],
"path.home": "/usr/share/elasticsearch",
"path.logs": "/var/log/elasticsearch",
"path.repo": [
"/mnt/nfs-share"
],
"path.shared_data": "",
"plugin.mandatory": [],
"readiness.port": "-1",
"reindex.remote.whitelist": [],
"remote_cluster.bind_host": [
"0.0.0.0"
],
"remote_cluster.host": [
"0.0.0.0"
],
"remote_cluster.max_request_header_size": "64kb",
"remote_cluster.port": "9443",
"remote_cluster.publish_host": [
"0.0.0.0"
],
"remote_cluster.publish_port": "-1",
"remote_cluster.tcp.keep_alive": "true",
"remote_cluster.tcp.keep_count": "-1",
"remote_cluster.tcp.keep_idle": "-1",
"remote_cluster.tcp.keep_interval": "-1",
"remote_cluster.tcp.no_delay": "true",
"remote_cluster.tcp.reuse_address": "true",
"remote_cluster.tcp.send_buffer_size": "-1b",
"remote_cluster_server.enabled": "false",
"repositories.fs.chunk_size": "9223372036854775807b",
"repositories.fs.location": "",
"repositories.url.allowed_urls": [],
"repositories.url.supported_protocols": [
"http",
"https",
"ftp",
"file",
"jar"
],
"repositories.url.url": "http:",
"repository_s3.compare_and_exchange.anti_contention_delay": "1s",
"repository_s3.compare_and_exchange.time_to_live": "15s",
"resource.reload.enabled": "true",
"resource.reload.interval.high": "5s",
"resource.reload.interval.low": "60s",
"resource.reload.interval.medium": "30s",
"rest.action.multi.allow_explicit_index": "true",
"runtime_fields.grok.watchdog.interval": "1s",
"runtime_fields.grok.watchdog.max_execution_time": "1s",
"script.allowed_contexts": [],
"script.allowed_types": [],
"script.cache.expire": "0ms",
"script.cache.max_size": "3000",
"script.disable_max_compilations_rate": "false",
"script.max_compilations_rate": "150/5m",
"script.max_size_in_bytes": "65535",
"script.painless.regex.enabled": "limited",
"script.painless.regex.limit-factor": "6",
"search.aggs.rewrite_to_filter_by_filter": "true",
"search.aggs.tdigest_execution_hint": "DEFAULT",
"search.allow_expensive_queries": "true",
"search.check_ccs_compatibility": "false",
"search.default_allow_partial_results": "true",
"search.default_keep_alive": "5m",
"search.default_search_timeout": "-1",
"search.highlight.term_vector_multi_value": "true",
"search.keep_alive_interval": "1m",
"search.low_level_cancellation": "true",
"search.max_async_search_response_size": "10mb",
"search.max_buckets": "65536",
"search.max_keep_alive": "24h",
"search.max_open_scroll_context": "500",
"search.query_phase_parallel_collection_enabled": "false",
"search.worker_threads_enabled": "true",
"searchable_snapshots.blob_cache.periodic_cleanup.batch_size": "100",
"searchable_snapshots.blob_cache.periodic_cleanup.interval": "1h",
"searchable_snapshots.blob_cache.periodic_cleanup.pit_keep_alive": "10m",
"searchable_snapshots.blob_cache.periodic_cleanup.retention_period": "1h",
"security.manager.filter_bad_defaults": "true",
"slm.health.failed_snapshot_warn_threshold": "5",
"slm.history_index_enabled": "true",
"slm.minimum_interval": "15m",
"slm.retention_duration": "1h",
"slm.retention_schedule": "0 30 1 * * ?",
"snapshot.max_concurrent_operations": "1000",
"snapshot.refresh_repo_uuid_on_restore": "true",
"stack.templates.enabled": "true",
"telemetry.metrics.enabled": "false",
"thread_pool.analyze.queue_size": "16",
"thread_pool.analyze.size": "1",
"thread_pool.auto_complete.queue_size": "100",
"thread_pool.auto_complete.size": "1",
"thread_pool.azure_event_loop.core": "0",
"thread_pool.azure_event_loop.keep_alive": "30s",
"thread_pool.azure_event_loop.max": "1",
"thread_pool.cluster_coordination.queue_size": "-1",
"thread_pool.cluster_coordination.size": "1",
"thread_pool.estimated_time_interval": "200ms",
"thread_pool.estimated_time_interval.warn_threshold": "5s",
"thread_pool.fetch_shard_started.core": "1",
"thread_pool.fetch_shard_started.keep_alive": "5m",
"thread_pool.fetch_shard_started.max": "8",
"thread_pool.fetch_shard_store.core": "1",
"thread_pool.fetch_shard_store.keep_alive": "5m",
"thread_pool.fetch_shard_store.max": "8",
"thread_pool.flush.core": "1",
"thread_pool.flush.keep_alive": "5m",
"thread_pool.flush.max": "2",
"thread_pool.force_merge.queue_size": "-1",
"thread_pool.force_merge.size": "1",
"thread_pool.generic.core": "4",
"thread_pool.generic.keep_alive": "30s",
"thread_pool.generic.max": "128",
"thread_pool.get.queue_size": "1000",
"thread_pool.get.size": "7",
"thread_pool.management.core": "1",
"thread_pool.management.keep_alive": "5m",
"thread_pool.management.max": "4",
"thread_pool.profiling.core": "0",
"thread_pool.profiling.keep_alive": "30m",
"thread_pool.profiling.max": "1",
"thread_pool.refresh.core": "1",
"thread_pool.refresh.keep_alive": "5m",
"thread_pool.refresh.max": "2",
"thread_pool.repository_azure.core": "0",
"thread_pool.repository_azure.keep_alive": "30s",
"thread_pool.repository_azure.max": "5",
"thread_pool.scheduler.warn_threshold": "5s",
"thread_pool.search.queue_size": "1000",
"thread_pool.search.size": "7",
"thread_pool.search_coordination.queue_size": "1000",
"thread_pool.search_coordination.size": "2",
"thread_pool.search_throttled.queue_size": "100",
"thread_pool.search_throttled.size": "1",
"thread_pool.search_worker.queue_size": "-1",
"thread_pool.search_worker.size": "7",
"thread_pool.snapshot.core": "1",
"thread_pool.snapshot.keep_alive": "5m",
"thread_pool.snapshot.max": "10",
"thread_pool.snapshot_meta.core": "1",
"thread_pool.snapshot_meta.keep_alive": "30s",
"thread_pool.snapshot_meta.max": "12",
"thread_pool.system_critical_read.queue_size": "2000",
"thread_pool.system_critical_read.size": "2",
"thread_pool.system_critical_write.queue_size": "1500",
"thread_pool.system_critical_write.size": "2",
"thread_pool.system_read.queue_size": "2000",
"thread_pool.system_read.size": "2",
"thread_pool.system_write.queue_size": "1000",
"thread_pool.system_write.size": "2",
"thread_pool.warmer.core": "1",
"thread_pool.warmer.keep_alive": "5m",
"thread_pool.warmer.max": "2",
"thread_pool.write.queue_size": "10000",
"thread_pool.write.size": "4",
"time_series.poll_interval": "5m",
"tracing.apm.enabled": "false",
"tracing.apm.names.exclude": [],
"tracing.apm.names.include": [],
"tracing.apm.sanitize_field_names": [
"password",
"passwd",
"pwd",
"secret",
"*key",
"*token*",
"*session*",
"*credit*",
"*card*",
"*auth*",
"*principal*",
"set-cookie"
],
"transport.bind_host": [
"0.0.0.0"
],
"transport.compress": "INDEXING_DATA",
"transport.compression_scheme": "LZ4",
"transport.connect_timeout": "30s",
"transport.connections_per_node.bulk": "3",
"transport.connections_per_node.ping": "1",
"transport.connections_per_node.recovery": "2",
"transport.connections_per_node.reg": "6",
"transport.connections_per_node.state": "1",
"transport.host": [
"0.0.0.0"
],
"transport.netty.boss_count": "1",
"transport.netty.receive_predictor_max": "64kb",
"transport.netty.receive_predictor_min": "64kb",
"transport.netty.receive_predictor_size": "64kb",
"transport.netty.worker_count": "4",
"transport.ping_schedule": "-1",
"transport.port": "9301",
"transport.publish_host": [
"0.0.0.0"
],
"transport.publish_port": "-1",
"transport.rst_on_close": "false",
"transport.slow_operation_logging_threshold": "5s",
"transport.tcp.keep_alive": "true",
"transport.tcp.keep_count": "-1",
"transport.tcp.keep_idle": "-1",
"transport.tcp.keep_interval": "-1",
"transport.tcp.no_delay": "true",
"transport.tcp.receive_buffer_size": "-1b",
"transport.tcp.reuse_address": "true",
"transport.tcp.send_buffer_size": "-1b",
"transport.tracer.exclude": [
"internal:coordination/fault_detection/*"
],
"transport.tracer.include": [],
"transport.type": "security4",
"transport.type.default": "netty4",
"write_load_forecaster.max_index_age": "7d",
"xpack.applications.behavioral_analytics.ingest.bulk_processor.flush_delay": "10s",
"xpack.applications.behavioral_analytics.ingest.bulk_processor.max_bytes_in_flight": "5%",
"xpack.applications.behavioral_analytics.ingest.bulk_processor.max_events_per_bulk": "500",
"xpack.applications.behavioral_analytics.ingest.bulk_processor.max_number_of_retries": "1",
"xpack.applications.rules.max_rules_per_ruleset": "100",
"xpack.autoscaling.memory.monitor.timeout": "15s",
"xpack.ccr.ccr_thread_pool.queue_size": "100",
"xpack.ccr.ccr_thread_pool.size": "32",
"xpack.ccr.enabled": "true",
"xpack.downsample.thread_pool.queue_size": "256",
"xpack.downsample.thread_pool.size": "1",
"xpack.ent_search.enabled": "true",
"xpack.eql.enabled": "true",
"xpack.graph.enabled": "true",
"xpack.http.connection_pool_ttl": "-1",
"xpack.http.default_connection_timeout": "10s",
"xpack.http.default_read_timeout": "10s",
"xpack.http.max_response_size": "10mb",
"xpack.http.proxy.host": "",
"xpack.http.proxy.port": "0",
"xpack.http.proxy.scheme": "",
"xpack.http.tcp.keep_alive": "true",
"xpack.http.whitelist": [
"*"
],
"xpack.idp.allowed_nameid_formats": [
"urn:oasis:names:tc:SAML:2.0:nameid-format:transient"
],
"xpack.idp.contact.email": "",
"xpack.idp.contact.given_name": "",
"xpack.idp.contact.surname": "",
"xpack.idp.defaults.authn_expiry": "5m",
"xpack.idp.defaults.nameid_format": "urn:oasis:names:tc:SAML:2.0:nameid-format:transient",
"xpack.idp.enabled": "false",
"xpack.idp.entity_id": "",
"xpack.idp.metadata.signing.keystore.alias": "",
"xpack.idp.organization.display_name": "",
"xpack.idp.organization.name": "",
"xpack.idp.organization.url": "http:",
"xpack.idp.privileges.application": "",
"xpack.idp.privileges.cache.size": "100",
"xpack.idp.privileges.cache.ttl": "90m",
"xpack.idp.signing.keystore.alias": "",
"xpack.idp.slo_endpoint.post": "https:",
"xpack.idp.slo_endpoint.redirect": "https:",
"xpack.idp.sp.cache.size": "1000",
"xpack.idp.sp.cache.ttl": "60m",
"xpack.idp.sp.wildcard.path": "wildcard_services.json",
"xpack.idp.sso_endpoint.post": "https:",
"xpack.idp.sso_endpoint.redirect": "https:",
"xpack.inference.utility_thread_pool.core": "0",
"xpack.inference.utility_thread_pool.keep_alive": "10m",
"xpack.inference.utility_thread_pool.max": "1",
"xpack.license.self_generated.type": "basic",
"xpack.license.upload.types": [
"standard",
"gold",
"platinum",
"enterprise",
"trial"
],
"xpack.ml.allocated_processors_scale": "1",
"xpack.ml.autodetect_process": "true",
"xpack.ml.datafeed_thread_pool.core": "1",
"xpack.ml.datafeed_thread_pool.keep_alive": "1m",
"xpack.ml.datafeed_thread_pool.max": "512",
"xpack.ml.delayed_data_check_freq": "15m",
"xpack.ml.enable_config_migration": "true",
"xpack.ml.enabled": "true",
"xpack.ml.inference_model.cache_size": "40%",
"xpack.ml.inference_model.time_to_live": "5m",
"xpack.ml.job_comms_thread_pool.core": "4",
"xpack.ml.job_comms_thread_pool.keep_alive": "1m",
"xpack.ml.job_comms_thread_pool.max": "2048",
"xpack.ml.max_anomaly_records": "500",
"xpack.ml.max_inference_processors": "50",
"xpack.ml.max_lazy_ml_nodes": "0",
"xpack.ml.max_machine_memory_percent": "30",
"xpack.ml.max_ml_node_size": "0b",
"xpack.ml.max_model_memory_limit": "0b",
"xpack.ml.max_open_jobs": "512",
"xpack.ml.min_disk_space_off_heap": "5gb",
"xpack.ml.model_repository": "https://ml-models.elastic.co",
"xpack.ml.native_inference_comms_thread_pool.core": "3",
"xpack.ml.native_inference_comms_thread_pool.keep_alive": "1m",
"xpack.ml.native_inference_comms_thread_pool.max": "312",
"xpack.ml.nightly_maintenance_requests_per_second": "-1.0",
"xpack.ml.node_concurrent_job_allocations": "2",
"xpack.ml.persist_results_max_retries": "20",
"xpack.ml.process_connect_timeout": "10s",
"xpack.ml.use_auto_machine_memory_percent": "false",
"xpack.ml.utility_thread_pool.core": "1",
"xpack.ml.utility_thread_pool.keep_alive": "10m",
"xpack.ml.utility_thread_pool.max": "2048",
"xpack.monitoring.collection.ccr.stats.timeout": "10s",
"xpack.monitoring.collection.cluster.stats.timeout": "10s",
"xpack.monitoring.collection.enrich.stats.timeout": "10s",
"xpack.monitoring.collection.index.recovery.active_only": "false",
"xpack.monitoring.collection.index.recovery.timeout": "10s",
"xpack.monitoring.collection.index.stats.timeout": "10s",
"xpack.monitoring.collection.indices": [],
"xpack.monitoring.collection.interval": "10s",
"xpack.monitoring.collection.ml.job.stats.timeout": "10s",
"xpack.monitoring.collection.node.stats.timeout": "10s",
"xpack.monitoring.elasticsearch.collection.enabled": "true",
"xpack.monitoring.history.duration": "168h",
"xpack.monitoring.migration.decommission_alerts": "false",
"xpack.monitoring.templates.enabled": "true",
"xpack.notification.email.account.domain_allowlist": [
"*"
],
"xpack.notification.email.default_account": "",
"xpack.notification.email.html.sanitization.allow": [
"body",
"head",
"_tables",
"_links",
"_blocks",
"_formatting",
"img:embedded"
],
"xpack.notification.email.html.sanitization.disallow": [],
"xpack.notification.email.html.sanitization.enabled": "true",
"xpack.notification.jira.default_account": "",
"xpack.notification.pagerduty.default_account": "",
"xpack.notification.reporting.interval": "15s",
"xpack.notification.reporting.retries": "40",
"xpack.notification.reporting.warning.enabled": "true",
"xpack.notification.slack.default_account": "",
"xpack.notification.webhook.additional_token_enabled": "false",
"xpack.profiling.check_outdated_indices": "true",
"xpack.profiling.enabled": "true",
"xpack.profiling.query.details.max_slices": "16",
"xpack.profiling.query.realtime": "true",
"xpack.profiling.query.stacktrace.max_slices": "16",
"xpack.profiling.templates.enabled": "false",
"xpack.rollup.task_thread_pool.queue_size": "-1",
"xpack.rollup.task_thread_pool.size": "1",
"xpack.searchable.snapshot.allocate_on_rolling_restart": "false",
"xpack.searchable.snapshot.cache.range_size": "32mb",
"xpack.searchable.snapshot.cache.recovery_range_size": "128kb",
"xpack.searchable.snapshot.cache.sync.interval": "60s",
"xpack.searchable.snapshot.cache.sync.max_files": "10000",
"xpack.searchable.snapshot.cache.sync.shutdown_timeout": "10s",
"xpack.searchable.snapshot.shared_cache.count_reads": "true",
"xpack.searchable.snapshot.shared_cache.decay.interval": "60s",
"xpack.searchable.snapshot.shared_cache.max_freq": "100",
"xpack.searchable.snapshot.shared_cache.min_time_delta": "60s",
"xpack.searchable.snapshot.shared_cache.mmap": "false",
"xpack.searchable.snapshot.shared_cache.range_size": "16mb",
"xpack.searchable.snapshot.shared_cache.recovery_range_size": "128kb",
"xpack.searchable.snapshot.shared_cache.region_size": "16mb",
"xpack.searchable.snapshot.shared_cache.size": "0",
"xpack.searchable.snapshot.shared_cache.size.max_headroom": "-1",
"xpack.searchable_snapshots.cache_fetch_async_thread_pool.core": "0",
"xpack.searchable_snapshots.cache_fetch_async_thread_pool.keep_alive": "30s",
"xpack.searchable_snapshots.cache_fetch_async_thread_pool.max": "12",
"xpack.searchable_snapshots.cache_prewarming_thread_pool.core": "0",
"xpack.searchable_snapshots.cache_prewarming_thread_pool.keep_alive": "30s",
"xpack.searchable_snapshots.cache_prewarming_thread_pool.max": "16",
"xpack.security.audit.enabled": "false",
"xpack.security.audit.logfile.emit_cluster_name": "false",
"xpack.security.audit.logfile.emit_cluster_uuid": "true",
"xpack.security.audit.logfile.emit_node_host_address": "false",
"xpack.security.audit.logfile.emit_node_host_name": "false",
"xpack.security.audit.logfile.emit_node_id": "true",
"xpack.security.audit.logfile.emit_node_name": "false",
"xpack.security.audit.logfile.events.emit_request_body": "false",
"xpack.security.audit.logfile.events.exclude": [],
"xpack.security.audit.logfile.events.include": [
"ACCESS_DENIED",
"ACCESS_GRANTED",
"ANONYMOUS_ACCESS_DENIED",
"AUTHENTICATION_FAILED",
"CONNECTION_DENIED",
"TAMPERED_REQUEST",
"RUN_AS_DENIED",
"RUN_AS_GRANTED",
"SECURITY_CONFIG_CHANGE"
],
"xpack.security.authc.anonymous.authz_exception": "true",
"xpack.security.authc.anonymous.roles": [],
"xpack.security.authc.anonymous.username": "_anonymous",
"xpack.security.authc.api_key.cache.hash_algo": "ssha256",
"xpack.security.authc.api_key.cache.max_keys": "25000",
"xpack.security.authc.api_key.cache.ttl": "24h",
"xpack.security.authc.api_key.delete.interval": "24h",
"xpack.security.authc.api_key.delete.retention_period": "7d",
"xpack.security.authc.api_key.delete.timeout": "-1",
"xpack.security.authc.api_key.doc_cache.ttl": "5m",
"xpack.security.authc.api_key.enabled": "true",
"xpack.security.authc.api_key.hashing.algorithm": "PBKDF2",
"xpack.security.authc.password_hashing.algorithm": "BCRYPT",
"xpack.security.authc.reserved_realm.enabled": "true",
"xpack.security.authc.run_as.enabled": "true",
"xpack.security.authc.service_token.cache.hash_algo": "ssha256",
"xpack.security.authc.service_token.cache.max_tokens": "100000",
"xpack.security.authc.service_token.cache.ttl": "20m",
"xpack.security.authc.success_cache.enabled": "true",
"xpack.security.authc.success_cache.expire_after_access": "1h",
"xpack.security.authc.success_cache.size": "10000",
"xpack.security.authc.token.delete.interval": "30m",
"xpack.security.authc.token.delete.timeout": "-1",
"xpack.security.authc.token.enabled": "false",
"xpack.security.authc.token.thread_pool.queue_size": "1000",
"xpack.security.authc.token.thread_pool.size": "1",
"xpack.security.authc.token.timeout": "20m",
"xpack.security.authz.store.privileges.cache.max_size": "10000",
"xpack.security.authz.store.privileges.cache.ttl": "24h",
"xpack.security.authz.store.roles.cache.max_size": "10000",
"xpack.security.authz.store.roles.field_permissions.cache.max_size_in_bytes": "104857600",
"xpack.security.authz.store.roles.has_privileges.cache.max_size": "1000",
"xpack.security.authz.store.roles.negative_lookup_cache.max_size": "10000",
"xpack.security.authz.timer.indices.enabled": "false",
"xpack.security.authz.timer.indices.threshold.debug": "20ms",
"xpack.security.authz.timer.indices.threshold.info": "100ms",
"xpack.security.authz.timer.indices.threshold.warn": "200ms",
"xpack.security.autoconfiguration.enabled": "true",
"xpack.security.automata.cache.enabled": "true",
"xpack.security.automata.cache.size": "10000",
"xpack.security.automata.cache.ttl": "48h",
"xpack.security.automata.max_determinized_states": "100000",
"xpack.security.crypto.thread_pool.queue_size": "1000",
"xpack.security.crypto.thread_pool.size": "2",
"xpack.security.dls.bitset.cache.size": "10%",
"xpack.security.dls.bitset.cache.ttl": "2h",
"xpack.security.dls_fls.enabled": "true",
"xpack.security.enabled": "true",
"xpack.security.encryption.algorithm": "AES/CTR/NoPadding",
"xpack.security.encryption_key.algorithm": "AES",
"xpack.security.encryption_key.length": "128",
"xpack.security.enrollment.enabled": "false",
"xpack.security.filter.always_allow_bound_address": "true",
"xpack.security.fips_mode.enabled": "false",
"xpack.security.http.filter.allow": [],
"xpack.security.http.filter.deny": [],
"xpack.security.http.filter.enabled": "true",
"xpack.security.http.ssl.enabled": "true",
"xpack.security.operator_privileges.enabled": "false",
"xpack.security.remote_cluster.filter.allow": [],
"xpack.security.remote_cluster.filter.deny": [],
"xpack.security.remote_cluster_client.ssl.enabled": "true",
"xpack.security.remote_cluster_server.ssl.enabled": "true",
"xpack.security.ssl.diagnose.trust": "true",
"xpack.security.transport.filter.allow": [],
"xpack.security.transport.filter.deny": [],
"xpack.security.transport.filter.enabled": "true",
"xpack.security.transport.ssl.enabled": "true",
"xpack.security.user": null,
"xpack.transform.num_transform_failure_retries": "10",
"xpack.transform.transform_scheduler_frequency": "1s",
"xpack.watcher.actions.bulk.default_timeout": "",
"xpack.watcher.actions.index.default_timeout": "",
"xpack.watcher.bulk.actions": "1",
"xpack.watcher.bulk.concurrent_requests": "0",
"xpack.watcher.bulk.flush_interval": "1s",
"xpack.watcher.bulk.size": "1mb",
"xpack.watcher.enabled": "true",
"xpack.watcher.encrypt_sensitive_data": "false",
"xpack.watcher.execution.default_throttle_period": "5s",
"xpack.watcher.execution.scroll.size": "0",
"xpack.watcher.execution.scroll.timeout": "",
"xpack.watcher.index.rest.direct_access": "",
"xpack.watcher.input.search.default_timeout": "",
"xpack.watcher.internal.ops.bulk.default_timeout": "",
"xpack.watcher.internal.ops.index.default_timeout": "",
"xpack.watcher.internal.ops.search.default_timeout": "",
"xpack.watcher.stop.timeout": "30s",
"xpack.watcher.thread_pool.queue_size": "1000",
"xpack.watcher.thread_pool.size": "1",
"xpack.watcher.transform.search.default_timeout": "",
"xpack.watcher.trigger.schedule.ticker.tick_interval": "500ms",
"xpack.watcher.use_ilm_index_management": "true",
"xpack.watcher.watch.scroll.size": "0"
}
}
}
I currently have this value set to:
"cluster.routing.allocation.balance.disk_usage": "2.0E-11",
What value would you recommend setting?
You can use the PUT _cluster/settings API to set that setting value to 1.0.
Do I understand correctly that you need to execute the request:
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.balance.disk_usage": 1.0
}
}
???
now GET _cluster/settings
{
"persistent": {
"xpack": {
"monitoring": {
"collection": {
"enabled": "true"
}
}
}
},
"transient": {}
}
thanks
Yup, you should start seeing shards being moved around.
thanks, Now the situation has improved a little. It looks like this:
I understand that the situation will not improve? from 150 to 279 GB
It really depends how your shard size distribution is like. If you have unbalanced shard size, then that will be the best you can get.
Сan you describe in simple words on what principle shards and indexes are distributed between nodes? According to the size of the fragments or their number? Since I’m constantly looking, I don’t have enough space on one node, and the rest are filled more or less evenly. In templates I usually use 2 or 4 shards.
now I have problem cluster status "yellow":
{
"decider": "disk_threshold",
"decision": "NO",
"explanation": "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low.max_headroom=200gb], having less than the minimum required [200gb] free space, actual free: [169.8gb], actual used: [88.4%]"
}
and
node shards disk.indices disk.used disk.avail disk.total disk.percent
v-elk-ed02 172 1tb 1.2tb 168.9gb 1.4tb 88
v-elk-ed03 743 1006.1gb 1tb 398.8gb 1.4tb 72
v-elk-ed01 743 1006.8gb 1tb 398.2gb 1.4tb 73
v-elk-ed04 658 1000.5gb 1tb 404.6gb 1.4tb 72
UNASSIGNED 4
How can I move shards by hand? or does it not make any sense? on all other nodes there is another 400 GB of free space, but on this 02 it is constantly on the verge
How large are your largest indices? How many shards do these have? Do you have an excessive amount of these allocated to the v-elk-ed02 node?
hi,
We mostly use 4 shards, less often 2 shards 1 replica. as an example:
index shard prirep state docs store dataset ip node
fs-winlog-min-2023.12.19 3 p STARTED 22766633 7.3gb 7.3gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.27 0 p STARTED 7879602 2.8gb 2.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.27 1 p STARTED 7880659 2.8gb 2.8gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-login-2023.09 0 p STARTED 21620538 8.6gb 8.6gb 10.16.5.207 v-elk-ed02
my77-ucsyslog-2024.02 0 p STARTED 23930884 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
my77-ucsyslog-2024.02 1 r STARTED 23926273 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
.enrich-service-itsm-scom_policy2-1702291401499 0 r STARTED 1818 232.2kb 232.2kb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.07 2 p STARTED 23146161 6.3gb 6.3gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.07 3 p STARTED 23153260 6.3gb 6.3gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12_part 0 p STARTED 142772625 43.9gb 43.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12_part 2 p STARTED 142805430 43.7gb 43.7gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12_part 3 p STARTED 142772189 43.6gb 43.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.02 0 p STARTED 9952876 4.3gb 4.3gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.16 0 p STARTED 15773208 4.8gb 4.8gb 10.16.5.207 v-elk-ed02
filebeat-erp-tgb-v-dvf-jkupn10-2024.02 1 p STARTED 46387432 14.9gb 14.9gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik03-2024.02 0 r STARTED 2870895 4.5gb 4.5gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik04-2024.02 1 p STARTED 2870897 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik06-2024.02 0 p STARTED 2850112 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik05-2024.02 1 p STARTED 2852018 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.06 0 p STARTED 17317916 6.7gb 6.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.06 1 p STARTED 17319428 6.7gb 6.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.06 2 p STARTED 17318077 6.7gb 6.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.06 3 p STARTED 17315193 6.7gb 6.7gb 10.16.5.207 v-elk-ed02
.enrich-service-itsm-scom_policy3-1710216001646 0 r UNASSIGNED
exchange-v-dfhn-ik03-2024.03 0 r STARTED 963723 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.29 0 p STARTED 13759706 5.2gb 5.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.29 2 p STARTED 13762261 5.2gb 5.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.29 3 p STARTED 13760778 5.2gb 5.2gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.27 2 p STARTED 29828398 9.6gb 9.6gb 10.16.5.207 v-elk-ed02
scom-srv-performance-2024.03 0 p STARTED 6716075 1.3gb 1.3gb 10.16.5.207 v-elk-ed02
scom-srv-performance-2024.03 1 p STARTED 6713107 1.3gb 1.3gb 10.16.5.207 v-elk-ed02
.monitoring-logstash-7-2024.03.06 0 r STARTED 11099840 1.7gb 1.7gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik03-2024.02 1 p STARTED 3442589 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
ngfw-block_corp-2024.03.12 0 p STARTED 3830418 2.1gb 2.1gb 10.16.5.207 v-elk-ed02
ngfw-block_corp-2024.03.12 1 p STARTED 3835641 2.1gb 2.1gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.07 0 r STARTED 7768633 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.07 1 r STARTED 7770987 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.05 0 p STARTED 7713667 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.05 1 p STARTED 7711291 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
.enrich-user-itsm-to-worksrtation_policy-1710216424733 0 r UNASSIGNED
fs-winlog-2024.01.26 1 p STARTED 11494708 5gb 5gb 10.16.5.207 v-elk-ed02
.enrich-service-itsm-scom-wan_policy-1702291396696 0 r STARTED 1778 732.6kb 732.6kb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.24 1 p STARTED 14640708 5.9gb 5.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.24 2 p STARTED 14642932 5.9gb 5.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.24 3 p STARTED 14635500 5.9gb 5.9gb 10.16.5.207 v-elk-ed02
my78-ngfw-logs-2024.03.12 0 p STARTED 15357191 9.6gb 9.6gb 10.16.5.207 v-elk-ed02
exchange-v-dfhn-ik03-2024.02 0 p STARTED 3116061 4.8gb 4.8gb 10.16.5.207 v-elk-ed02
exchange-v-tujn-mn03-2024.02 1 r STARTED 3114365 4.8gb 4.8gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.03-new 0 r STARTED 6811009 2.1gb 2.1gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.03-new 1 r STARTED 6809977 2.1gb 2.1gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.11 0 p STARTED 17928852 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.11 1 p STARTED 17921865 6.1gb 6.1gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.11 2 p STARTED 17923558 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2024.02 0 p STARTED 1503284 562.3mb 562.3mb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2024.02 1 p STARTED 1505029 560mb 560mb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.31 0 p STARTED 14274498 3.9gb 3.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.31 1 p STARTED 14279931 3.9gb 3.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.31 2 p STARTED 14281505 4gb 4gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.31 3 p STARTED 14281024 4gb 4gb 10.16.5.207 v-elk-ed02
exchange-v-exch-fe03-2024.03 0 r STARTED 1152602 1.9gb 1.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.20 1 p STARTED 17157685 5.9gb 5.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.25 0 p STARTED 11936311 3.5gb 3.5gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.25 1 p STARTED 11940034 3.5gb 3.5gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.25 2 p STARTED 11933966 3.5gb 3.5gb 10.16.5.207 v-elk-ed02
ftp-logs-2023.05 1 p STARTED 65886090 9.1gb 9.1gb 10.16.5.207 v-elk-ed02
ngfw-block_corp-2024.03.09 0 p STARTED 6200291 3.1gb 3.1gb 10.16.5.207 v-elk-ed02
ngfw-block_corp-2024.03.09 1 p STARTED 6201501 3.1gb 3.1gb 10.16.5.207 v-elk-ed02
ngfw-block_corp-2024.03.09 2 p STARTED 6199340 3.1gb 3.1gb 10.16.5.207 v-elk-ed02
exchange-v-exch-fe02-2024.02 0 r STARTED 3466776 5.5gb 5.5gb 10.16.5.207 v-elk-ed02
exchange-v-exch-fe02-2024.02 1 p STARTED 3467269 5.5gb 5.5gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.11 0 p STARTED 22224591 21.6gb 21.6gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.11 1 p STARTED 22221427 21.6gb 21.6gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-geo-2023.06 0 p STARTED 7782933 3gb 3gb 10.16.5.207 v-elk-ed02
.enrich-user-workstation_policy-1702291423716 0 r STARTED 14945 762.5kb 762.5kb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.29 0 p STARTED 20725852 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.29 3 p STARTED 20733859 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
filebeat-hjj-aud-v-rnu-perpn55-2024.03 0 r STARTED 10097015 2.1gb 2.1gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.06 2 p STARTED 23722486 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
.monitoring-logstash-7-2024.03.08 0 p STARTED 11123504 1.7gb 1.7gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.08 2 p STARTED 43537630 15.9gb 15.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.08 3 p STARTED 43543578 15.8gb 15.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.15 1 p STARTED 11132703 5.1gb 5.1gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.15 2 p STARTED 11138909 5.1gb 5.1gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.07 2 p STARTED 15314092 5.6gb 5.6gb 10.16.5.207 v-elk-ed02
ngfw-block-2024.02 0 p STARTED 3187955 1.2gb 1.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.14 0 p STARTED 6406676 2.2gb 2.2gb 10.16.5.207 v-elk-ed02
sftp-logs-2023.06 1 p STARTED 3044754 496.5mb 496.5mb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.06 3 p STARTED 17156355 6.1gb 6.1gb 10.16.5.207 v-elk-ed02
.enrich-network_policy-1710216153396 0 r UNASSIGNED
fs-winlog-2024.02.10 1 p STARTED 14309526 3.5gb 3.5gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.31 0 p STARTED 16868425 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.31 1 p STARTED 16867192 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.31 3 p STARTED 16871728 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.12 3 p STARTED 23861914 7.6gb 7.6gb 10.16.5.207 v-elk-ed02
.monitoring-logstash-7-2024.03.09 0 p STARTED 11085298 1.7gb 1.7gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-login-2024.02 0 p STARTED 16354188 5.8gb 5.8gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-login-2024.02 1 p STARTED 16354433 5.8gb 5.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.20 0 p STARTED 7829633 2.8gb 2.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.20 2 p STARTED 7831263 2.8gb 2.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.01 0 p STARTED 13597493 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.01 1 p STARTED 13597698 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.01 2 p STARTED 13595497 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.01 3 p STARTED 13598897 6.2gb 6.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.14 0 p STARTED 18793673 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.14 1 p STARTED 18791955 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.14 2 p STARTED 18791916 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.14 3 p STARTED 18790975 6.5gb 6.5gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.03 0 p STARTED 8428053 2.9gb 2.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.03 1 p STARTED 8427834 2.9gb 2.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.17 3 p STARTED 13038265 5.4gb 5.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.29 0 p STARTED 12842982 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.29 1 p STARTED 12854429 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.29 2 p STARTED 12850419 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.29 3 p STARTED 12847273 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.08 0 p STARTED 15430595 5gb 5gb 10.16.5.207 v-elk-ed02
.monitoring-es-7-2024.03.12 0 p STARTED 3381449 1.6gb 1.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.17 0 p STARTED 20179180 5.6gb 5.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.17 2 p STARTED 20184385 5.6gb 5.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.17 3 p STARTED 20172512 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.26 1 p STARTED 21323908 7.2gb 7.2gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.16 0 p STARTED 11923338 5.6gb 5.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.05 2 p STARTED 21640863 7.6gb 7.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.05 3 p STARTED 21635062 7.6gb 7.6gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.04-new 0 p STARTED 2622042 837.6mb 837.6mb 10.16.5.207 v-elk-ed02
winlogbeat-srv-user-workstation-2023.04-new 1 r STARTED 2621311 850.4mb 850.4mb 10.16.5.207 v-elk-ed02
winlogbeat-srv-login-2023.04 0 p STARTED 18172958 7.4gb 7.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.17 1 p STARTED 13260382 2.9gb 2.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.17 2 p STARTED 13259929 2.9gb 2.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.17 3 p STARTED 13256654 2.9gb 2.9gb 10.16.5.207 v-elk-ed02
ftp-logs-2024.01 0 p STARTED 66868410 9.5gb 9.5gb 10.16.5.207 v-elk-ed02
ftp-logs-2024.01 1 p STARTED 66890587 9.5gb 9.5gb 10.16.5.207 v-elk-ed02
pexip-syslog-2024.03 0 p STARTED 1715168 1004.6mb 1004.6mb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.12 0 p STARTED 16710458 5.6gb 5.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.03 0 p STARTED 14138850 4gb 4gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.06 0 p STARTED 17750837 6.5gb 6.5gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.06 1 p STARTED 17759663 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.06 2 p STARTED 17758895 6.6gb 6.6gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.13 2 p STARTED 16295318 5.9gb 5.9gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.04 1 p STARTED 23888628 8.4gb 8.4gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2024.01.04 2 p STARTED 23898922 8.4gb 8.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.05 0 p STARTED 12283757 5.4gb 5.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.05 3 p STARTED 12287337 5.4gb 5.4gb 10.16.5.207 v-elk-ed02
scom-srv-performance-2023.12 1 p STARTED 14110420 2.8gb 2.8gb 10.16.5.207 v-elk-ed02
scom-srv-performance-2023.07 0 p STARTED 19977606 3.9gb 3.9gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.12 0 p STARTED 12125137 12.2gb 12.2gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.12 1 p STARTED 12127455 11.6gb 11.6gb 10.16.5.207 v-elk-ed02
exchange-v-exch-fe05-2024.03 0 p STARTED 987113 2.7gb 2.7gb 10.16.5.207 v-elk-ed02
rdweb-info-2024 0 p STARTED 2721776 804.9mb 804.9mb 10.16.5.207 v-elk-ed02
rdweb-info-2024 1 r STARTED 2721814 791.9mb 791.9mb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.12.14 1 p STARTED 22667037 7.1gb 7.1gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.09 0 p STARTED 13722879 13.6gb 13.6gb 10.16.5.207 v-elk-ed02
winlogbeat-srv-sdc-2024.03.09 1 p STARTED 13727072 13.6gb 13.6gb 10.16.5.207 v-elk-ed02
filebeat-uap-v-yap-fgh01-2024.02 0 p STARTED 30164448 9.6gb 9.6gb 10.16.5.207 v-elk-ed02
filebeat-uap-v-yap-fgt01-2024.02 1 p STARTED 30167771 9.6gb 9.6gb 10.16.5.207 v-elk-ed02
exchange-v-etuo-cc01-2024.02 0 p STARTED 1484113 2.4gb 2.4gb 10.16.5.207 v-elk-ed02
exchange-v-etuo-cc01-2024.02 1 p STARTED 1488051 2.3gb 2.3gb 10.16.5.207 v-elk-ed02
scom-srv-performance-2024.01 0 p STARTED 17557018 3.5gb 3.5gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.25 0 p STARTED 17810307 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.25 1 p STARTED 17806223 4.4gb 4.4gb 10.16.5.207 v-elk-ed02
fs-winlog-min-2023.10 1 p STARTED 25995392 9.8gb 9.8gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.23 2 p STARTED 11634968 5.4gb 5.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.19 1 p STARTED 9904107 4.5gb 4.5gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.19 3 p STARTED 9906810 4.5gb 4.5gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.18 0 p STARTED 13950622 3.4gb 3.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.02.18 1 p STARTED 13950683 3.4gb 3.4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.02 1 p STARTED 14154167 4gb 4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.02 2 p STARTED 14167623 4gb 4gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.03.02 3 p STARTED 14165002 4gb 4gb 10.16.5.207 v-elk-ed02
ftp-logs-2023.12 0 p STARTED 68847167 9.7gb 9.7gb 10.16.5.207 v-elk-ed02
ftp-logs-2023.11 0 p STARTED 47315162 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
ftp-logs-2023.11 1 p STARTED 47300321 6.9gb 6.9gb 10.16.5.207 v-elk-ed02
fs-winlog-2024.01.30 2 p STARTED 12555915 5.7gb 5.7gb 10.16.5.207 v-elk-ed02
filebeat-nap-v-gjk-webput7-2024.03 0 p STARTED 4063326 904.1mb 904.1mb 10.16.5.207 v-elk-ed02
filebeat-nap-v-gjh-webput7-2024.03 1 p STARTED 4061369 893.9mb 893.9mb 10.16.5.207 v-elk-ed02
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.