Hey everyone!
We are running Elasticsearch 5.5.3, with architecture of 3 master and 2 data nodes.
I have upgrade the kernel from 4.9.62-21.56 to 4.14.55-62.37 running on amazon Linux 2018.03.
After rebooting the ec2 instance I have looked on the logs and catch something strange,
one of the data node was writing to log "no known master node, scheduling a retry" and then the node find a master, but still print log "no known master node, scheduling a retry".
The node seems to work OK, cluster in Green state and everything seems to work just as expected. I was wondering why it prompt "no known master" right after it found one.
Your help is much appreciated!
Note: for the sake of security, I removed IPs from log file.
[2018-07-29T10:54:58,114][INFO ][o.e.n.Node ] [] initializing ...
[2018-07-29T10:54:58,185][INFO ][o.e.e.NodeEnvironment ] [qAysMHX] using [1] data paths, mounts [[/ (/dev/nvme0n1p1)]], net usable_space [42.6gb], net total_space [157.3gb], spins? [no], types [ext4]
[2018-07-29T10:54:58,185][INFO ][o.e.e.NodeEnvironment ] [qAysMHX] heap size [7.9gb], compressed ordinary object pointers [true]
[2018-07-29T10:54:58,687][INFO ][o.e.n.Node ] node name [qAysMHX] derived from node ID [qAysMHXfRNKN0MEC9jnSpw]; set [node.name] to override
[2018-07-29T10:54:58,687][INFO ][o.e.n.Node ] version[5.5.3], pid[2219], build[9305a5e/2017-09-07T15:56:59.599Z], OS[Linux/4.14.55-62.37.amzn1.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2018-07-29T10:54:58,687][INFO ][o.e.n.Node ] JVM arguments [-Xms8g, -Xmx8g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [aggs-matrix-stats]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [ingest-common]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [lang-expression]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [lang-groovy]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [lang-mustache]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [lang-painless]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [parent-join]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [percolator]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [reindex]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [transport-netty3]
[2018-07-29T10:55:00,291][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded module [transport-netty4]
[2018-07-29T10:55:00,292][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded plugin [discovery-ec2]
[2018-07-29T10:55:00,292][INFO ][o.e.p.PluginsService ] [qAysMHX] loaded plugin [repository-s3]
[2018-07-29T10:55:01,839][INFO ][o.e.d.DiscoveryModule ] [qAysMHX] using discovery type [ec2]
[2018-07-29T10:55:02,461][INFO ][o.e.n.Node ] initialized
[2018-07-29T10:55:02,461][INFO ][o.e.n.Node ] [qAysMHX] starting ...
[2018-07-29T10:55:02,607][INFO ][o.e.t.TransportService ] [qAysMHX] publish_address {IP:9300}, bound_addresses {[::]:9300}
[2018-07-29T10:55:02,614][INFO ][o.e.b.BootstrapChecks ] [qAysMHX] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-07-29T10:55:03,148][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:03,151][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:03,803][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:04,232][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:04,604][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:04,695][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:05,216][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:05,585][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:05,587][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:06,355][INFO ][o.e.c.s.ClusterService ] [qAysMHX] detected_master {i6kYZl8}{i6kYZl8tTgmolB_XB_S3Ag}{bUh9t8XjRWefqcMyviNNFw}{IP}{IP:9300}, added {{RwW4p2s}{RwW4p2swTi6_v6cuYdWJTw}{m7mIv7KhR_Cj4UnVifRJqQ}{IP}{IP:9300},{i6kYZl8}{i6kYZl8tTgmolB_XB_S3Ag}{bUh9t8XjRWefqcMyviNNFw}{IP}{IP:9300},{yTNq9fL}{yTNq9fLeTNyHp4kkVHO9BA}{hNyGKI6gSHm0UIB0wpUA9A}{IP}{IP:9300},{sz7ZvOD}{sz7ZvODTSCO0VjU7Ul3SVw}{SGbXKOWbR1eA_ym4meTIQg}{IP}{IP:9300},}, reason: zen-disco-receive(from master [master {i6kYZl8}{i6kYZl8tTgmolB_XB_S3Ag}{bUh9t8XjRWefqcMyviNNFw}{IP}{IP:9300} committed version [404642]])
[2018-07-29T10:55:06,780][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:06,985][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [qAysMHX] no known master node, scheduling a retry
[2018-07-29T10:55:07,159][INFO ][o.e.h.n.Netty4HttpServerTransport] [qAysMHX] publish_address {IP:9200}, bound_addresses {[::]:9200}
[2018-07-29T10:55:07,160][INFO ][o.e.n.Node ] [qAysMHX] started