Kibana Master Not Discovered Exception

Mustafa_Akmal · October 29, 2018, 1:42pm

We have a single node cluster. Elastic Search is up and running. Getting this error
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

Basically Kibana cant discover master.
Kibanna UI:
Kibana%20Screenshot
Versions:
Elastic Search: 5.6.2
Kibana: 5.6.2

What could the possible reason for this?

bhavyarm · October 29, 2018, 1:52pm

Hi,

I think that is an elasticsearch error? Can you please look at your es logs and kibana logs and post them?

How are you connecting to ES through your Kibana? Have you configured ES port and host in your Kibana.yml?

https://www.elastic.co/guide/en/kibana/5.2/settings.html

Thanks,
Bhavya

Mustafa_Akmal · October 30, 2018, 8:04am

Hey,
Thanks for the response.
I had a look at the logs and I think the problem is with Elastic Search. It takes the single node as a data node rather than a master.

metrondemo.log
[2018-10-30T07:48:50,023][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:48:50,716][INFO ][o.e.n.Node [2018-10-30T07:48:50,721][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:48:50,729][INFO ][o.e.n.Node [2018-10-30T07:48:50,729][INFO ][o.e.n.Node [2018-10-30T07:48:50,737][INFO ][o.e.n.Node [2018-10-30T07:48:53,965][INFO ][o.e.n.Node [2018-10-30T07:48:54,046][INFO ][o.e.e.NodeEnvironment [2018-10-30T07:48:54,046][INFO ][o.e.e.NodeEnvironment [2018-10-30T07:48:54,047][INFO ][o.e.n.Node [2018-10-30T07:48:54,048][INFO ][o.e.n.Node [2018-10-30T07:48:54,048][INFO ][o.e.n.Node [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,745][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,746][INFO ][o.e.p.PluginsService [2018-10-30T07:48:54,746][INFO ][o.e.p.PluginsService [2018-10-30T07:48:55,948][INFO ][o.e.d.DiscoveryModule [2018-10-30T07:48:56,374][INFO ][o.e.n.Node [2018-10-30T07:48:56,374][INFO ][o.e.n.Node [2018-10-30T07:48:56,497][INFO ][o.e.t.TransportService [2018-10-30T07:48:56,507][INFO ][o.e.b.BootstrapChecks [2018-10-30T07:49:06,532][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:49:16,533][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:49:26,525][WARN ][o.e.n.Node [2018-10-30T07:49:26,535][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:49:26,537][INFO ][o.e.h.n.Netty4HttpSe [2018-10-30T07:49:26,537][INFO ][o.e.n.Node [2018-10-30T07:49:36,535][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:49:46,536][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:49:56,537][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:50:06,538][WARN ][o.e.d.z.ZenDiscovery [2018-10-30T07:50:16,539][WARN ][o.e.d.z.ZenDiscovery ] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] stopping ...
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] stopped
] [metron.abcdata.org] closing ...
] [metron.abcdata.org] closed
] [metron.abcdata.org] initializing ...
] [metron.abcdata.org] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [286.5gb], net total_space [299.9gb], spins? [unknown], types [rootfs]
] [metron.abcdata.org] heap size [989.8mb], compressed ordinary object pointers [true]
] [metron.abcdata.org] node name [metron.abcdata.org], node ID [HXvWyogRTY2uuLaLE4M6gA]
] [metron.abcdata.org] version[5.6.2], pid[5175], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Linux/3.10.0-862.3.2.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_112/25.112-b15]
] [metron.abcdata.org] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -verbose:gc, -Xloggc:/var/log/elasticsearch/elasticsearch_gc.log, -XX:-CMSConcurrentMTEnabled, -XX:+PrintGCDateStamps, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps, -XX:ErrorFile=/var/log/elasticsearch/elasticsearch_err.log, -XX:ParallelGCThreads=8, -Xms1024m, -Xmx1024m, -Des.path.home=/usr/share/elasticsearch]
] [metron.abcdata.org] loaded module [aggs-matrix-stats]
] [metron.abcdata.org] loaded module [ingest-common]
] [metron.abcdata.org] loaded module [lang-expression]
] [metron.abcdata.org] loaded module [lang-groovy]
] [metron.abcdata.org] loaded module [lang-mustache]
] [metron.abcdata.org] loaded module [lang-painless]
] [metron.abcdata.org] loaded module [parent-join]
] [metron.abcdata.org] loaded module [percolator]
] [metron.abcdata.org] loaded module [reindex]
] [metron.abcdata.org] loaded module [transport-netty3]
] [metron.abcdata.org] loaded module [transport-netty4]
] [metron.abcdata.org] no plugins loaded
] [metron.abcdata.org] using discovery type [zen]
] [metron.abcdata.org] initialized
] [metron.abcdata.org] starting ...
] [metron.abcdata.org] publish_address {172.31.20.116:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {172.31.20.116:9300}
] [metron.abcdata.org] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] timed out while waiting for initial discovery state - timeout: 30s
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
rverTransport] [metron.abcdata.org] publish_address {172.31.20.116:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {172.31.20.116:9200}
] [metron.abcdata.org] started
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again
] [metron.abcdata.org] not enough master nodes discovered during pinging (found [[]], but needed [-1]), pinging again

Mustafa_Akmal · October 30, 2018, 8:05am

elasticsearch.yml
cluster:
name: metrondemo
routing:
allocation.node_concurrent_recoveries: 4
allocation.disk.watermark.low: .97
allocation.disk.threshold_enabled: true
allocation.disk.watermark.high: 0.99

discovery:
zen:
ping:
unicast:
hosts: [metron.abcdata.org]

node:
data: true
master: false
name: metron.abcdata.org
path:
data: "/opt/lmm/es_data"

http:
port: 9200-9300
cors.enabled: "false"

transport:
tcp:
port: 9300-9400

gateway:
recover_after_data_nodes: 3
recover_after_time: 30s
expected_data_nodes: 0

https://www.elastic.co/guide/en/elasticsearch/guide/current/indexing-performance.html

indices:
store.throttle.type: none
memory:
index_buffer_size: 10%
fielddata:
cache.size: 25%

bootstrap:
memory_lock: true
system_call_filter: false

thread_pool:
bulk:
queue_size: 3000
index:
queue_size: 1000

discovery.zen.ping_timeout: 10s
discovery.zen.fd.ping_interval: 30s
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5

network.host: [ local, site ]
network.publish_host: []

The node category shows data as true and master as false. I think this is the where the problem is. How to fix this? Thanks!

system · November 27, 2018, 8:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.