Kibana Master Not Discovered Exception

[Mustafa_Akmal]

We have a single node cluster. Elastic Search is up and running. Getting this error
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Basically Kibana cant discover master.
Kibanna UI:

Versions:
Elastic Search: 5.6.2
Kibana: 5.6.2

What could the possible reason for this?

[bhavyarm]

Hi,

I think that is an elasticsearch error? Can you please look at your es logs and kibana logs and post them?

How are you connecting to ES through your Kibana? Have you configured ES port and host in your Kibana.yml?

https://www.elastic.co/guide/en/kibana/5.2/settings.html

Thanks,
Bhavya

[Mustafa_Akmal]

Hey,
Thanks for the response.
I had a look at the logs and I think the problem is with Elastic Search. It takes the single node as a data node rather than a master.

metrondemo.log
[2018-10-30T07:48:50,023][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:48:50,716][INFO ][o.e.n.Node ] [metron.abcdata.org] stopping ...
[2018-10-30T07:48:50,721][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:48:50,729][INFO ][o.e.n.Node ] [metron.abcdata.org] stopped
[2018-10-30T07:48:50,729][INFO ][o.e.n.Node ] [metron.abcdata.org] closing ...
[2018-10-30T07:48:50,737][INFO ][o.e.n.Node ] [metron.abcdata.org] closed
[2018-10-30T07:48:53,965][INFO ][o.e.n.Node ] [metron.abcdata.org] initializing ...
[2018-10-30T07:48:54,046][INFO ][o.e.e.NodeEnvironment ] [metron.abcdata.org] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [286.5gb], net total_space [299.9gb], spins? [unknown], types [rootfs]
[2018-10-30T07:48:54,046][INFO ][o.e.e.NodeEnvironment ] [metron.abcdata.org] heap size [989.8mb], compressed ordinary object pointers [true]
[2018-10-30T07:48:54,047][INFO ][o.e.n.Node ] [metron.abcdata.org] node name [metron.abcdata.org], node ID [HXvWyogRTY2uuLaLE4M6gA]
[2018-10-30T07:48:54,048][INFO ][o.e.n.Node ] [metron.abcdata.org] version[5.6.2], pid[5175], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/3.10.0-862.3.2.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_112/25.112-b15]
[2018-10-30T07:48:54,048][INFO ][o.e.n.Node ] [metron.abcdata.org] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -verbose:gc, -Xloggc:/var/log/elasticsearch/elasticsearch_gc.log, -XX:-CMSConcurrentMTEnabled, -XX:+PrintGCDateStamps, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps, -XX:ErrorFile=/var/log/elasticsearch/elasticsearch_err.log, -XX:ParallelGCThreads=8, -Xms1024m, -Xmx1024m, -Des.path.home=/usr/share/elasticsearch]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [aggs-matrix-stats]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [ingest-common]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [lang-expression]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [lang-groovy]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [lang-mustache]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [lang-painless]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [parent-join]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [percolator]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [reindex]
[2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [transport-netty3]
[2018-10-30T07:48:54,746][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] loaded module [transport-netty4]
[2018-10-30T07:48:54,746][INFO ][o.e.p.PluginsService ] [metron.abcdata.org] no plugins loaded
[2018-10-30T07:48:55,948][INFO ][o.e.d.DiscoveryModule ] [metron.abcdata.org] using discovery type [zen]
[2018-10-30T07:48:56,374][INFO ][o.e.n.Node ] [metron.abcdata.org] initialized
[2018-10-30T07:48:56,374][INFO ][o.e.n.Node ] [metron.abcdata.org] starting ...
[2018-10-30T07:48:56,497][INFO ][o.e.t.TransportService ] [metron.abcdata.org] publish_address {172.31.20.116:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {172.31.20.116:9300}
[2018-10-30T07:48:56,507][INFO ][o.e.b.BootstrapChecks ] [metron.abcdata.org] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-10-30T07:49:06,532][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:49:16,533][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:49:26,525][WARN ][o.e.n.Node ] [metron.abcdata.org] timed out while waiting for initial discovery state - timeout: 30s
[2018-10-30T07:49:26,535][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:49:26,537][INFO ][o.e.h.n.Netty4HttpServerTransport] [metron.abcdata.org] publish_address {172.31.20.116:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {172.31.20.116:9200}
[2018-10-30T07:49:26,537][INFO ][o.e.n.Node ] [metron.abcdata.org] started
[2018-10-30T07:49:36,535][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:49:46,536][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:49:56,537][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:50:06,538][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
[2018-10-30T07:50:16,539][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again

[Mustafa_Akmal]

elasticsearch.yml
cluster:
name: metrondemo
routing:
allocation.node_concurrent_recoveries: 4
allocation.disk.watermark.low: .97
allocation.disk.threshold_enabled: true
allocation.disk.watermark.high: 0.99

discovery:
zen:
ping:
unicast:
hosts: [metron.abcdata.org]

node:
data: true
master: false
name: metron.abcdata.org
path:
data: "/opt/lmm/es_data"

http:
port: 9200-9300
cors.enabled: "false"

transport:
tcp:
port: 9300-9400

gateway:
recover_after_data_nodes: 3
recover_after_time: 30s
expected_data_nodes: 0

https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html

indices:
store.throttle.type: none
memory:
index_buffer_size: 10%
fielddata:
cache.size: 25%

bootstrap:
memory_lock: true
system_call_filter: false

thread_pool:
bulk:
queue_size: 3000
index:
queue_size: 1000

discovery.zen.ping_timeout: 10s
discovery.zen.fd.ping_interval: 30s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5

network.host: [ local, site ]
network.publish_host: []

The node category shows data as true and master as false. I think this is the where the problem is. How to fix this? Thanks!