No monitoring data found: IndexNotFoundException

[vnallamothu (vamshi)]

Hi, I'm seeing "No monitoring data found in Kibana Dashboard". In elasticsearch logs, I found following:

[2018-11-05T11:11:09,553][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error whi le indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: [.monitoring-kibana -6-2018.11.05] IndexNotFoundException[no such index]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$th rowExportException$2(LocalBulk.java:130) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java: 193) ~[?:1.8.0_181]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java: 175) ~[?:1.8.0_181]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators .java:948) ~[?:1.8.0_181]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_181]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.ja va:471) ~[?:1.8.0_181]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.j ava:151) ~[?:1.8.0_181]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEac hOps.java:174) ~[?:1.8.0_181]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_181]
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418 ) ~[?:1.8.0_181]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExpo rtException(LocalBulk.java:131) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$do Flush$0(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.j ava:59) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.support.TransportAction$1.onResponse(Transpo rtAction.java:88) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.support.TransportAction$1.onResponse(Transpo rtAction.java:84) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier .lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:583) ~[elasticsear ch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.j ava:59) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun (TransportBulkAction.java:343) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(Abstrac tRunnable.java:37) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(Transpo rtBulkAction.java:474) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(Transport BulkAction.java:205) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkI ndexIngestRequest$3(TransportBulkAction.java:526) ~[elasticsearch-5.5.3.jar:5.5. 3]

Can any one help me?

[Aaron_Caldwell]

Hello Vamshi,

You may want to see Tim's response to this discuss question and see if it's of any help:

No monitoring data found error. The .monitoring index is also not found

I am seeing the error "No monitoring data found" since I've installed Kibana 6.2. Can anyone help me with this?

Of course, you may already be aware of our docs but you might want to double-check your settings here: Monitoring settings in Elasticsearch | Elasticsearch Guide [8.11] | Elastic

Regards,
Aaron

[vnallamothu (vamshi)]

Hi Aaron,

Thank you for the response. I've already looked into Tim's response earlier and I think it is for different usecase. What happened in this situation is, Developer deleted all the existing indexes and put action.auto_create_index to false and reindexed a new index.

After that, I was getting following errors:

[2018-11-05T11:38:58,574][WARN ][o.e.x.m.MonitoringService] [es-node01] monitoring execution failed
org.elasticsearch.xpack.monitoring.exporter.ExportException: Exception when closing export bulk
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1$1.(ExportBulk.java:106) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$1.onFailure(ExportBulk.java:104) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:217) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound$1.onResponse(ExportBulk.java:211) ~[?:?]
at org.elasticsearch.xpack.common.IteratingActionListener.onResponse(IteratingActionListener.java:108) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:175) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:137) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:114) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:88) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:84) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:583) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:343) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:474) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:205) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$3(TransportBulkAction.java:526) ~[elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:98) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.3.jar:5.5.3]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.3.jar:5.5.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: failed to flush export bulks
at org.elasticsearch.xpack.monitoring.exporter.ExportBulk$Compound.lambda$null$0(ExportBulk.java:167) ~[?:?]
... 19 more
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:126) ~[?:?]

can you please let me know if action.auto_create_index should be true? (we made to set it to false as bitbucket's documentation recommends it). Please let me know.