I just set up a GKE ES cluster with 3 master eligible nodes, 3 data nodes, and 2 coordinator nodes.
I haven't created any index yet.
Following is the logging from Stackdriver:
[2019-09-24 23:01:24.158 PDT]
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
- severity: "ERROR"
- textPayload: "OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-09-24 23:01:30.636 PDT]
version[7.3.0], pid[1], build[default/docker/de777fa/2019-07-24T18:30:11.767338Z], OS[Linux/4.14.127+/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12.0.1/12.0.1+12]
I noticed that every time this error occurred, the nodes will leave and rejoin, which happens about 8 ~ 10 times per day.
For example:
[2019-09-24 22:59:15.254 PDT]
node-left[{elasticsearch-pr-master-2}{WewdseokSy6AXRBE-ORPbw}{dAI_yzFcSVSdsO8Drc7QRg}{10.4.4.8}{10.4.4.8:9300}{m}{xpack.installed=true} disconnected], term: 3, version: 31, reason: removed {{elasticsearch-pr-master-2}{WewdseokSy6AXRBE-ORPbw}{dAI_yzFcSVSdsO8Drc7QRg}{10.4.4.8}{10.4.4.8:9300}{m}{xpack.installed=true},}
[2019-09-24 22:59:14.515 PDT]
"stacktrace": ["org.elasticsearch.transport.NodeDisconnectedException: [elasticsearch-pr-master-2][10.4.4.8:9300][disconnected] disconnected"] }
[2019-09-24 22:59:14.564 PDT]
"Caused by: java.net.ConnectException: Connection refused",
[2019-09-24 22:59:14.579 PDT]
"stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [elasticsearch-pr-master-2][10.4.4.8:9300] connect_exception",
[2019-09-24 22:59:14.579 PDT]
"Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.4.4.8/10.4.4.8:9300",
[2019-09-24 22:59:14.612 PDT]
"Caused by: java.net.ConnectException: Connection refused",
May I get help on how to fix this?
Thanks!