Issues in ES 2.2.0

Elastic Stack Elasticsearch
1

i am building production cluster and after ES 2.2 installation i am seeing issues as soon as i bring up Kibana, i have a similar configuration in lower environment and i don't see any issues like these, below is my configurations

  1. Elasticsearch.yml

cluster.name: support-prod
node.name: sd-1
path.data: /apps/data/esdata/data/
path.logs: /apps/data/esdata/logs/
node.master: true
node.data: false
bootstrap.memory_lock: true
network.host: ipselsprd01
network.bind_host: 0.0.0.0
network.publish_host: hostB
discovery.zen.ping.unicast.hosts: ["ipselsprd01","ipselsprd02"] -- commented for now
discovery.zen.minimum_master_nodes: 2 -- commented for now
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.watcher.enabled: false

  1. kibana.yml

server.host: 5601
server.host: "ipselsprd01"
server.name: "ipselsprd02-sp"
elasticsearch.url: "http://xxxxxx:9200"
kibana.index: "ipselsprd01.kibana"
logging.dest: /apps/data/kibana/kibana.log
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
xpack.reporting.enabled: false

  1. Elasticsearch log

[2017-03-06T11:47:16,243][WARN ][r.suppressed ] path: /ipselsprd01.kibana/config/_search, params: {index=ipselsprd01.kibana, type=config}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:208) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:115) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:154) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:53) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:173) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:145) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:87) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:75) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:64) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:530) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$1(RestSearchAction.java:81) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:82) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:162) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.HttpServer.dispatchRequest(HttpServer.java:115) [elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:515) [transport-netty4-5.2.0.jar:5.2.0]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:70) [transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:66) [transport-netty4-5.2.0.jar:5.2.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.7.Final.jar:4.1.7.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
at
......Caused by: org.elasticsearch.action.NoShardAvailableActionException
... 57 more

3

kibana logs

{"type":"log","@timestamp":"2017-03-06T16:43:47Z","tags":["status","plugin:kibana@5.2.0","info"],"pid":6539,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:elasticsearch@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:xpack_main@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:monitoring@5.2.0","info"],"pid":6539,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:searchprofiler@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:tilemap@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:console@5.2.0","info"],"pid":6539,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","plugin:timelion@5.2.0","info"],"pid":6539,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["listening","info"],"pid":6539,"message":"Server running at http://xxxxxx:5601"}
{"type":"log","@timestamp":"2017-03-06T16:43:48Z","tags":["status","ui settings","info"],"pid":6539,"state":"yellow","message":"Status changed from uninitialized to yellow - Elasticsearch plugin is yellow","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-06T16:43:53Z","tags":["status","plugin:xpack_main@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-03-06T16:43:53Z","tags":["status","plugin:searchprofiler@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-03-06T16:43:53Z","tags":["status","plugin:tilemap@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-03-06T16:43:53Z","tags":["status","plugin:elasticsearch@5.2.0","info"],"pid":6539,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

{"type":"log","@timestamp":"2017-03-06T16:44:23Z","tags":["status","plugin:searchprofiler@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from yellow to red - Unable to create Kibana index "ipselsprd01.kibana"","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2017-03-06T16:44:23Z","tags":["status","plugin:tilemap@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from yellow to red - Unable to create Kibana index "ipselsprd01.kibana"","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2017-03-06T16:44:23Z","tags":["status","plugin:elasticsearch@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from yellow to red - Unable to create Kibana index "ipselsprd01.kibana"","prevState":"yellow","prevMsg":"No existing Kibana index found"}
{"type":"log","@timestamp":"2017-03-06T16:44:23Z","tags":["status","ui settings","error"],"pid":6539,"state":"red","message":"Status changed from yellow to red - Elasticsearch plugin is red","prevState":"yellow","prevMsg":"Elasticsearch plugin is yellow"}
{"type":"log","@timestamp":"2017-03-06T16:44:26Z","tags":["status","plugin:xpack_main@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from red to red - [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Unable to create Kibana index "ipselsprd01.kibana""}
{"type":"log","@timestamp":"2017-03-06T16:44:26Z","tags":["status","plugin:searchprofiler@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from red to red - [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Unable to create Kibana index "ipselsprd01.kibana""}
{"type":"log","@timestamp":"2017-03-06T16:44:26Z","tags":["status","plugin:tilemap@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from red to red - [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Unable to create Kibana index "ipselsprd01.kibana""}
{"type":"log","@timestamp":"2017-03-06T16:44:26Z","tags":["status","plugin:elasticsearch@5.2.0","error"],"pid":6539,"state":"red","message":"Status changed from red to red - [search_phase_execution_exception] all shards failed","prevState":"red","prevMsg":"Unable to create Kibana index "ipselsprd01.kibana""}

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.