Unable to access elastic console

Ravi_Vishwakarma · November 14, 2022, 5:16pm

Hi,

we have already created space on Elasticsearch server but still facing issue.
we also checked for cluster allocation and found below output.

``{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index" : "logstash-2022.10.28-000283",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2022-11-14T09:51:53.196Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [OGDf2uTBSrqMUgxwdNucSg]: failed recovery, failure RecoveryFailedException[[logstash-2022.10.28-000283][0]: Recovery failed on {Prod-ELKPROD}{OGDf2uTBSrqMUgxwdNucSg}{fjOvLonZR16OrB6yF58W6Q}{172.19.1.125}{172.19.1.125:9300}{cdfhilmrstw}{ml.machine_memory=8139542528, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=4072669184}]; nested: IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/elk/elasticsearch/nodes/0/indices/FmuUNJTSTHuSiZ3QENg-rw/0/translog] is corrupted]; nested: NoSuchFileException[/elk/elasticsearch/nodes/0/indices/FmuUNJTSTHuSiZ3QENg-rw/0/translog/translog-6965.tlog]; ",
"last_allocation_status" : "no"
},
"can_allocate" : "yes",
"allocate_explanation" : "can allocate the shard",
"target_node" : {
"id" : "OGDf2uTBSrqMUgxwdNucSg",
"name" : "Prod-ELKPROD",
"transport_address" : "xxx.xxx.xx.xx:9300",
"attributes" : {
"ml.machine_memory" : "8139542528",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "4072669184"
}
},
"allocation_id" : "KsHiCPXCTViZuslB_y_t3A",
"node_allocation_decisions" : [
{
"node_id" : "OGDf2uTBSrqMUgxwdNucSg",
"node_name" : "Prod-ELKPROD",
"transport_address" : "xxx.xxx.xx.xx:9300",
"node_attributes" : {
"ml.machine_memory" : "8139542528",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "4072669184"
},
"node_decision" : "yes",
"store" : {
"in_sync" : true,
"allocation_id" : "KsHiCPXCTViZuslB_y_t3A"
}
}
]
}`

Elasticsearch logs below.

[2022-11-09T16:22:44,539][WARN ][o.e.x.m.e.l.LocalExporter] [Prod-ELKPROD] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[index [.monitoring-es-7-2022.11.09] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:135) ~[x-pack-monitoring-7.17.2.jar:7.17.2]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:136) [x-pack-monitoring-7.17.2.jar:7.17.2]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:117) [x-pack-monitoring-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:818) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.runRerouteP

Let me know how i can resolve this issue because at moment my ELK stack is down not able to access the Elasticsearch web console .

Thanks.

rashmi · November 14, 2022, 5:58pm

This error indicates a data node is critically low on disk space and has reached the flood-stage disk usage watermark. To prevent a full disk, when a node reaches this watermark, Elasticsearch blocks writes to any index with a shard on the node. If the block affects related system indices, Kibana and other Elastic Stack features may become unavailable.

Elasticsearch will automatically remove the write block when the affected node’s disk usage goes below the high disk watermark. To achieve this, Elasticsearch automatically moves some of the affected node’s shards to other nodes in the same data tier.

To immediately restore write operations, you can temporarily increase the disk watermarks and remove the write block.

Follow the instructions here: Fix watermark errors | Elasticsearch Guide [master] | Elastic

hope this resolves the issue.

Thanks
Rashmi

Ravi_Vishwakarma · November 15, 2022, 1:57pm

Hi Rashmi,

Thanks for the reply now we are able to access the elasticsearch console and all the index is also working fine except one index it is showing index status RED below is the error detail.

{
  "took": 28,
  "timed_out": false,
  "_shards": {
    "total": 242,
    "successful": 241,
    "skipped": 241,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "logstash-2022.10.28-000283",
        "node": null,
        "reason": {
          "type": "no_shard_available_action_exception",
          "reason": null,
          "index_uuid": "FmuUNJTSTHuSiZ3QENg-rw",
          "shard": "0",
          "index": "logstash-2022.10.28-000283"
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}

{
  "index.blocks.read_only_allow_delete": "false",
  "index.priority": "1",
  "index.query.default_field": [
    "*"
  ],
  "index.write.wait_for_active_shards": "1",
  "index.routing.allocation.include._tier_preference": "data_content",
  "index.routing.allocation.require._id": "OGDf2uTBSrqMUgxwdNucSg",
  "index.refresh_interval": "5s",
  "index.blocks.write": "true",
  "index.number_of_replicas": "1"
}

Please let me know how i can resolve this issue.

Regards,
Ravi

system · January 10, 2023, 1:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
UNASSIGNED ALLOCATION_FAILED Elasticsearch	2	623	February 10, 2023
Kibana_task_manager ALLOCATION_FAILED Elasticsearch docker	1	596	February 10, 2023
Unassigned Shard Elasticsearch	4	713	January 3, 2020
No_shard_available_action_exception Elasticsearch	1	14	November 12, 2024
Elasticsearch isn’t allowed to allocate this shard to any of the nodes in the cluster as a result logs are not store into elasticsearch Elasticsearch elastic-stack-monitoring , docker	4	237	April 22, 2024

Unable to access elastic console

Related topics