Unexpected error while indexing monitoring document org.elasticsearch.xpack.monitoring

Elastic Stack Elasticsearch
1

Hello,

I updated from 5.6.2 to 6.2.1 on Sunday and today I started to get this error, which stopped all the indexing on my nodes whic made some data to be lost since I have tcp and udp inputs comingo from logstash that are lost if not indexed.

[2018-02-20T21:01:10,064][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: UnavailableShardsException[[.monitoring-es-6-2018.02.21][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.monitoring-es-6-2018.02.21][0]] containing [233] requests]]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:140) ~[?:?]
        at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_161]
        at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_161]
        at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_161]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_161]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_161]
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_161]
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_161]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_161]
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_161]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:141) ~[?:?]
        at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:123) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.2.1.jar:6.2.1]
(a lot more of lines like this)

The error started at 21:00, local time, which corresponds to 00:00 UTC, when the new index are created and this error was getting repeated and only stopped after I restarted the elastic service on the nodes.

What should I do to avoid getting this problem again?

1 Like
2

I think that I found the problem, it seems that I mistakenly had leave the cluster.routing.allocation.enable set to none instead of all, when I restarted the nodes I set it to all.

2 Likes
3

Thanks for posting your solution!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.