Elasticsearch won't index any documents after installing X-Pack

Elastic Stack Elasticsearch
1

After installing X-Pack and restarting Elasticsearch I'm getting thousands of lines of exceptions in the Elasticsearch logs, and none of my input documents that I'm expecting are getting indexed.

I've got a vast number of these

[2017-08-09T15:00:06,618][DEBUG][o.e.a.b.TransportBulkAction] [et3-tim-monitor] failed to execute pipeline [xpack_monitoring_6] for document [.monitoring-es-6-2017.08.09/doc/null]
java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_6] does not exist
at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

followed by a vast number of these

[2017-08-09T15:00:06,892][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_6] does not exist
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:130) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_131]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_131]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_131]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_131]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[?:1.8.0_131]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[?:1.8.0_131]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_131]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[?:1.8.0_131]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:131) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:84) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:127) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.bulk.TransportBulkAction$IngestBulkResponseListener.onResponse(TransportBulkAction.java:624) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.bulk.TransportBulkAction$IngestBulkResponseListener.onResponse(TransportBulkAction.java:604) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$3(TransportBulkAction.java:524) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:98) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.1.jar:5.5.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_6] does not exist
at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.5.1.jar:5.5.1]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) ~[elasticsearch-5.5.1.jar:5.5.1]
... 5 more

(there may be other, different, stuff but I've sort-of lost interest wading through it).

I've found a similar issue where the solution was "completely reinstall Elasticsearch" but that's not exactly attractive, not least because it doesn't explain anything (and does it guarantee I keep all my existing data?).

2

It turns out that the exception messages stop after a while, and the problem is elsewhere other than in Elasticsearch.

It seems that when installing X-Pack you have to turn off sniffing in Logstash, but I didn't see that explained in the "how to install X-Pack" documentation.

Then you have to change the Logstash configuration further so that it quotes a user name and password to Elasticsearch, but again I didn't see that explained in the "how to install X-Pack" documentation. No doubt I'll have to do something similar to the beats that feed Elasticsearch without going through Logstash.

Suggestions for improvements:

(1) Remove or fix all causes of Java exception stack traces appearing in logs - the user should never see these except in the case of a coding error in the product, they should never be "business as usual".

(2) Improve the X-Pack installation instructions so that they include all the steps you actually need to take to get your working system working again after installing X-Pack.

(3) Improve error messages in the logs so that they actually give some clues as to what you're supposed to do about them.

(4) Consider defaulting security to off so that on installing X-Pack working systems continue working, and the user can then enable security if and only if they want to, and when and only when they're ready to do so.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.