Hi
We are not receiving the log from elasticsearch to kibana, also i have shared in below kibana and elasticsearch log. Can you please help to fix this issue.
service versions:
- logstash-6.3.2
- kafka_2.12-2.2.0
- zookeeper-3.4.14
- elasticsearch-6.3.2
- kibana-6.3.2
- filebeat-6.3.2
Kibana Log:
May 13 14:08:53 ssc-kibana kibana: {"type":"error","@timestamp":"2019-05-13T18:08:53Z","tags":["warning","monitoring-ui","kibana-monitoring"],"pid":31226,"level":"error","error":{"message":"[search_phase_execution_exception] all shards failed","name":"Error","stack":"[search_phase_execution_exception] all shards failed :: {"path":"/.reporting-*/_search","query":{"filter_path":"hits.total,aggregations.jobTypes.buckets,aggregations.objectTypes.buckets,aggregations.layoutTypes.buckets,aggregations.statusTypes.buckets"},"body":"{\"size\":0,\"aggs\":{\"jobTypes\":{\"terms\":{\"field\":\"jobtype\",\"size\":2}},\"objectTypes\":{\"terms\":{\"field\":\"meta.objectType.keyword\",\"size\":3}},\"layoutTypes\":{\"terms\":{\"field\":\"meta.layout.keyword\",\"size\":3}},\"statusTypes\":{\"terms\":{\"field\":\"status\",\"size\":4}}}}","statusCode":503,"response":"{\"error\":{\"root_cause\":,\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":},\"status\":503}"}\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:307:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:266:7)\n at HttpConnector. (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)\n at IncomingMessage.bound (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)\n at emitNone (events.js:111:20)\n at IncomingMessage.emit (events.js:208:7)\n at endReadableNT (_stream_readable.js:1064:12)\n at _combinedTickCallback (internal/process/next_tick.js:138:11)\n at process._tickDomainCallback (internal/process/next_tick.js:218:9)"},"message":"[search_phase_execution_exception] all shards failed"}
Elasticsearch Log:
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:210) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:189) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2019-05-13T14:39:04,349][WARN ][r.suppressed ] path: /.kibana/doc/config%3A6.3.2, params: {index=.kibana, id=config:6.3.2, type=doc}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.3.2]: routing [null]]
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:143) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:128) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:137) [transport-netty4-6.3.2.jar:6.3.2]
