Kibana server is not ready yet because upgrading elasticsearch

hi everyone i have a problem after i upgrade from 7.15.2 to 7.16.2 my kibana server is not ready yet.

here' s the log i get

[root@bdi-uat-els elasticsearch]# tail elasticsearch.log
        at org.elasticsearch.xpack.monitoring.MonitoringService$MonitoringExecution$1.doRun(MonitoringService.java:269) [x-pack-monitoring-7.16.2.jar:7.16.2]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.2.jar:7.16.2]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.2.jar:7.16.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2021-12-22T13:49:21,479][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.102:47860}
[2021-12-22T13:49:25,836][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:48612}

and when i type systemctl status Elasticsearch -l

[root@bdi-uat-els ansible]# systemctl status elasticsearch -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2021-12-21 09:53:24 WIB; 1 day 3h ago
Docs: [https://www.elastic.co](https://www.elastic.co/)
Main PID: 24106 (java)
CGroup: /system.slice/elasticsearch.service
├─24106 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseG1GC -[Djava.io](http://djava.io/).tmpdir=/tmp/elasticsearch-9138365068025045118 -XX:+HeapDumpOnOutOfMemoryError -XX:+ExitOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Dlog4j2.formatMsgNoLookups=true -Xms3910m -Xmx3910m -XX:MaxDirectMemorySize=2049966080 -XX:G1HeapRegionSize=4m -XX:InitiatingHeapOccupancyPercent=30 -XX:G1ReservePercent=15 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
└─24309 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

have you updated kibana as well?

yes i update everything

could you check connectivity and kibana logs , and share as much info as you can?

1 Like

how to check the connectivity?

see the linked topic.

what should i do?

"name" : "bdi-uat-els",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "TvUzG6h7T2-xEPKIUr1gjg",
"version" : {
"number" : "7.16.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "2b937c44140b6559905130a8650c64dbd0879cfb",
"build_date" : "2021-12-18T19:42:46.604893745Z",
"build_snapshot" : false,
"lucene_version" : "8.10.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"

this kibana log

{"type":"log","@timestamp":"2021-12-23T10:56:25+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:28+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:30+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:33+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:35+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:38+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:40+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:43+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:45+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
{"type":"log","@timestamp":"2021-12-23T10:56:48+07:00","tags":["info","elasticsearch","deprecation"],"pid":4068,"message":"Elasticsearch deprecation: 299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb \"transport.publish_address was printed as [ip:port] instead of [hostname/ip:port]. This format is deprecated and will change to [hostname/ip:port] in a future version. Use -Des.transport.cname_in_publish_address=true to enforce non-deprecated formatting.\"\nOrigin:kibana\nQuery:\n200\nGET /_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip"}
1 Like

Please post your kibana.yml file. I noticed an exception in the Elasticsearch logs, can you also post the full exception log.

1 Like

how i see the full exception log Elasticsearch?

about my config sorry that's a lot of confidential in there. there's no change. i have update from 7.10 to 7.15 and there's no problem. but why when i update 7.15.2 to 7.16.2 i got this error?

so no solution?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.