I noticed yesterday that my Metricbeat service had stopped running on the Windows test box I have. When I start Logstash I receive the below information.
[2021-06-18T10:28:12,009][WARN ][logstash.outputs.elasticsearch][main][6d1545737824a8dbf91a10409724d61c6b8801e5bb7ba7c85d905a3ade6ef5f3] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.13.0-2021.06.18", :routing=>nil}, #LogStash::Event:0x18c4a449], :response=>{"index"=>{"_index"=>"winlogbeat-7.13.0-2021.06.18", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}}}}
In my startup of Kibana I receive the following:
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
log [10:34:06.171] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster
log [10:34:16.158] [warning][kibana-monitoring][monitoring][monitoring][plugins] ResponseError: export_exception
at onBody (D:\ELK\kibana-7.12.1-windows-x86_64\node_modules@elastic\elasticsearch\lib\Transport.js:337:23)
at IncomingMessage.onEnd (D:\ELK\kibana-7.12.1-windows-x86_64\node_modules@elastic\elasticsearch\lib\Transport.js:264:11)
at IncomingMessage.emit (events.js:327:22)
at endReadableNT (internal/streams/readable.js:1327:12)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
log [10:34:16.159] [warning][kibana-monitoring][monitoring][monitoring][plugins] Unable to bulk upload the stats payload to the local cluster
And in ElasticSearch I get:
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$applyInternal$2(SecurityActionFilter.java:165) [x-pack-security-7.12.1.jar:7.12.1]
at org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:167) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.ActionListener$5.onResponse(ActionListener.java:286) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.ActionListener$MappedActionListener.onResponse(ActionListener.java:76) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:117) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:498) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:644) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.action.bulk.TransportBulkAction$1$2.doRun(TransportBulkAction.java:282) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) [elasticsearch-7.12.1.jar:7.12.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.12.1.jar:7.12.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.xpack.monitoring.exporter.ExportException: bulk [default_local] reports failures when exporting documents
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:122) ~[?:?]
... 20 more
I have copied over a fresh version of both the Kibana.yml and EalsticSearch.yml files and receive the same messages, I am unable to start the metricbeat service as it times out. I have made no changes to my configuration over the past two weeks as I have been working on locating information only.
Also when I open a PowerShell and try to run a .\metricbeat test config I get an error on line 98, did not find expected key; which is a blank line in my configuration. See below metricbeat config file.
Should I try a fresh metricbeat.yml file?