Elastic memmory size

Elastic Stack Elasticsearch
1

Hi i have this error when i ingest my nodes with logstsh data the thing is that the nodes were block because the memory what to small for the size of data but i change the path data to another that has 5 Teras o f memory increase the hep memory to half it has 48g memory what i need to solvei have try several ways to unblock the nodes. but also my cluster is down i can restarted

[2023-07-26T15:28:17,335][WARN ][o.e.x.m.e.l.LocalExporter] [elkdatanodek14] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: org.elasticsearch.cluster.block.ClusterBlockException: index [.monitoring-es-7-2023.07.26] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179) ~[?:?]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:1006) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:110) ~[?:?]
        at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:158) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onResponse(NodeClient.java:160) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:205) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:199) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) ~[?:?]
        at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:151) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:210) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:628) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:623) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.client.internal.node.NodeClient$SafelyWrappedActionListener.onFailure(NodeClient.java:170) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:217) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:59) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:71) ~[elasticsearch-8.8.2.jar:?]
        at org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27) ~[elasticsearch-8.8.2.jar:?]
2

Dear @hlcxpl ,

Please ```
PUT my_index/_settings
{ "index": { "index.blocks.read_only" : false } }

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-blocks.html

hope it helps.
and also please set /etc/elasticsearch/jvm.options to half of the memory which has been assigned.

Best Regards
3

but my entired lcuster is down how can i set thi without the devtool

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.