Unable to get Elasticsearch console

Ravi_Vishwakarma · November 9, 2022, 11:37am

Hi,
We are not able to get Elasticsearch web console even if we check https://xxx.xxx.xx.xxx:9200 it shows nothing but all the services are running fine.

Please find below logs from Elasticsearch.

[2022-11-09T16:22:44,539][WARN ][o.e.x.m.e.l.LocalExporter] [Prod-ELKPROD] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[index [.monitoring-es-7-2022.11.09] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:135) ~[x-pack-monitoring-7.17.2.jar:7.17.2]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:136) [x-pack-monitoring-7.17.2.jar:7.17.2]
	at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:117) [x-pack-monitoring-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:818) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.2.jar:7.17.2]
	at org.elasticsearch.action.support.replication.TransportReplicationAction.runRerouteP

leandrojmp · November 9, 2022, 12:36pm

How many nodes do you have? This log means that the disk in at least one of the nodes is full, you need to clean space by deleting some indices or other things in the server.

Ravi_Vishwakarma · November 9, 2022, 1:46pm

Hi,

We are having only one node. And regarding deleting the index can you please suggest me how i can delete the index because i am not able to get Elasticsearch web console. only i can access Elasticsearch server through putty.

leandrojmp · November 9, 2022, 2:07pm

You will need to make requests directly to Elasticsearch from the server or a machine that can access it.

Check the documentation on how to list the indices.

Example:

curl -X GET "localhost:9200/_cat/indices/my-index-*?v=true&s=index&pretty"

After you identify any index that you can delete, you can use the DELETE API.

Example:

curl -X DELETE "localhost:9200/my-index-000001?pretty"

Ravi_Vishwakarma · November 10, 2022, 7:05am

Hi,

Not able to run the above command it give below error.
Please find below error snaps.

leandrojmp · November 10, 2022, 12:59pm

Hello,

Avoid sharing screenshots of text, share the plaintext using the preformatted option in the editor, the </>, use screenshots only for visual errors.

From what I was able to see in your screenshot, you didn't pass the credentials, you need to use curl with -u, not -U.

Ravi_Vishwakarma · November 11, 2022, 5:40am

Hi,

Also we are getting below error when we checked in Elasticsearch logs.

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionExcetion: Search rejected due to missing shards [[.kibana_7.17.2_001][0]]. Consider using 'allow_partial
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:227) ~[elasticsearch-7.17.2.jar:7.17.2]

Thanks

Ravi_Vishwakarma · November 14, 2022, 2:08pm

Hi,

we have created space on Elasticsearch server but still facing issue.
we also checked for cluster allocation and found below output.

``{
"note" : "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
"index" : "logstash-2022.10.28-000283",
"shard" : 0,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2022-11-14T09:51:53.196Z",
"failed_allocation_attempts" : 5,
"details" : "failed shard on node [OGDf2uTBSrqMUgxwdNucSg]: failed recovery, failure RecoveryFailedException[[logstash-2022.10.28-000283][0]: Recovery failed on {Prod-ELKPROD}{OGDf2uTBSrqMUgxwdNucSg}{fjOvLonZR16OrB6yF58W6Q}{172.19.1.125}{172.19.1.125:9300}{cdfhilmrstw}{ml.machine_memory=8139542528, xpack.installed=true, transform.node=true, ml.max_open_jobs=512, ml.max_jvm_size=4072669184}]; nested: IndexShardRecoveryException[failed recovery]; nested: TranslogCorruptedException[translog from source [/elk/elasticsearch/nodes/0/indices/FmuUNJTSTHuSiZ3QENg-rw/0/translog] is corrupted]; nested: NoSuchFileException[/elk/elasticsearch/nodes/0/indices/FmuUNJTSTHuSiZ3QENg-rw/0/translog/translog-6965.tlog]; ",
"last_allocation_status" : "no"
},
"can_allocate" : "yes",
"allocate_explanation" : "can allocate the shard",
"target_node" : {
"id" : "OGDf2uTBSrqMUgxwdNucSg",
"name" : "Prod-ELKPROD",
"transport_address" : "xxx.xxx.xx.xx:9300",
"attributes" : {
"ml.machine_memory" : "8139542528",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "4072669184"
}
},
"allocation_id" : "KsHiCPXCTViZuslB_y_t3A",
"node_allocation_decisions" : [
{
"node_id" : "OGDf2uTBSrqMUgxwdNucSg",
"node_name" : "Prod-ELKPROD",
"transport_address" : "xxx.xxx.xx.xx:9300",
"node_attributes" : {
"ml.machine_memory" : "8139542528",
"xpack.installed" : "true",
"transform.node" : "true",
"ml.max_open_jobs" : "512",
"ml.max_jvm_size" : "4072669184"
},
"node_decision" : "yes",
"store" : {
"in_sync" : true,
"allocation_id" : "KsHiCPXCTViZuslB_y_t3A"
}
}
]
}`


Elasticsearch logs below.

[2022-11-09T16:22:44,539][WARN ][o.e.x.m.e.l.LocalExporter] [Prod-ELKPROD] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[index [.monitoring-es-7-2022.11.09] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:135) ~[x-pack-monitoring-7.17.2.jar:7.17.2]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:136) [x-pack-monitoring-7.17.2.jar:7.17.2]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:117) [x-pack-monitoring-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:82) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:31) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:192) [x-pack-security-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$DelegatingFailureActionListener.onResponse(ActionListener.java:219) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$RunBeforeActionListener.onResponse(ActionListener.java:389) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:101) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:625) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:620) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:97) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:38) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:1041) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:818) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.2.jar:7.17.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.runRerouteP

Let me know how i can resolve this issue because at moment my ELK stack is down not able to access the Elasticsearch web console .

Thanks.

system · December 12, 2022, 2:09pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.