hi everyone, i have a kibana. Yesterday I was still able to login, but today I can't. it only says "oops! Error. Try again" the username and password are all correct and the elsticsearch and kibana services are running properly. please help thank you
You will really need to provide more information please.
What version?
What is the entire error?
What do the Kibana logs show?
What do the Elasticsearch logs show?
i used v7.10
in login screen kibana "oops! Error. Try again
kibana log
Sep 8 13:50:31 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:50:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:51:00 bid-uat-kbn filebeat: 2021-09-08T13:51:00.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926940,"time":{"ms":6}},"total":{"ticks":1936900,"time":{"ms":8},"value":1936900},"user":{"ticks":1009960,"time":{"ms":2}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817560060}},"memstats":{"gc_next":20837456,"memory_alloc":11086208,"memory_total":127462598800},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.1,"15":0.05,"5":0.04,"norm":{"1":0.025,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:51:01 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:51:01Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:51:30 bid-uat-kbn filebeat: 2021-09-08T13:51:30.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926950,"time":{"ms":6}},"total":{"ticks":1936920,"time":{"ms":9},"value":1936920},"user":{"ticks":1009970,"time":{"ms":3}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817590061}},"memstats":{"gc_next":20837456,"memory_alloc":11420656,"memory_total":127462933248},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.06,"15":0.05,"5":0.04,"norm":{"1":0.015,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:51:31 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:51:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:52:00 bid-uat-kbn filebeat: 2021-09-08T13:52:00.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926950,"time":{"ms":8}},"total":{"ticks":1936920,"time":{"ms":16},"value":1936920},"user":{"ticks":1009970,"time":{"ms":8}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817620061}},"memstats":{"gc_next":20806640,"memory_alloc":10530288,"memory_total":127463377384},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.04,"15":0.05,"5":0.04,"norm":{"1":0.01,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:52:01 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:52:01Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:52:30 bid-uat-kbn filebeat: 2021-09-08T13:52:30.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926960,"time":{"ms":7}},"total":{"ticks":1936930,"time":{"ms":7},"value":1936930},"user":{"ticks":1009970}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817650062}},"memstats":{"gc_next":20806640,"memory_alloc":10748320,"memory_total":127463595416},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.02,"15":0.05,"5":0.03,"norm":{"1":0.005,"15":0.0125,"5":0.0075}}}}}}
Sep 8 13:52:31 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:52:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:53:00 bid-uat-kbn filebeat: 2021-09-08T13:53:00.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926960,"time":{"ms":3}},"total":{"ticks":1936940,"time":{"ms":9},"value":1936940},"user":{"ticks":1009980,"time":{"ms":6}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817680060}},"memstats":{"gc_next":20806640,"memory_alloc":11075904,"memory_total":127463923000},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.08,"15":0.05,"5":0.05,"norm":{"1":0.02,"15":0.0125,"5":0.0125}}}}}}
Sep 8 13:53:01 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:53:01Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:53:26 bid-uat-kbn filebeat: 2021-09-08T13:53:26.700+0700#011INFO#011log/harvester.go:333#011File is inactive: /home/ansible/journal. Closing because close_inactive of 5m0s reached.
Sep 8 13:53:30 bid-uat-kbn filebeat: 2021-09-08T13:53:30.949+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926970,"time":{"ms":6}},"total":{"ticks":1936950,"time":{"ms":8},"value":1936950},"user":{"ticks":1009980,"time":{"ms":2}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":10},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817710060}},"memstats":{"gc_next":20806640,"memory_alloc":11421776,"memory_total":127464268872},"runtime":{"goroutines":23}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"closed":1,"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.05,"15":0.05,"5":0.04,"norm":{"1":0.0125,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:53:31 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:53:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:53:31 bid-uat-kbn filebeat: 2021-09-08T13:53:31.700+0700#011INFO#011log/harvester.go:302#011Harvester started for file: /home/ansible/journal
Sep 8 13:53:34 bid-uat-kbn systemd-logind: New session 39681 of user ansible.
Sep 8 13:53:34 bid-uat-kbn systemd-logind: New session 39681 of user ansible.
Sep 8 13:53:34 bid-uat-kbn systemd: Started Session 39681 of user ansible.
Sep 8 13:53:39 bid-uat-kbn su: (to root) ansible on pts/0
Sep 8 13:53:39 bid-uat-kbn su: (to root) ansible on pts/0
Sep 8 13:54:00 bid-uat-kbn filebeat: 2021-09-08T13:54:00.950+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926980,"time":{"ms":9}},"total":{"ticks":1936970,"time":{"ms":19},"value":1936970},"user":{"ticks":1009990,"time":{"ms":10}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817740062}},"memstats":{"gc_next":20841712,"memory_alloc":10432968,"memory_total":127464668672},"runtime":{"goroutines":28}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":0.03,"15":0.05,"5":0.04,"norm":{"1":0.0075,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:54:01 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:54:01Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
Sep 8 13:54:30 bid-uat-kbn filebeat: 2021-09-08T13:54:30.950+0700#011INFO#011[monitoring]#011log/log.go:145#011Non-zero metrics in the last 30s#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":926980,"time":{"ms":6}},"total":{"ticks":1936980,"time":{"ms":11},"value":1936980},"user":{"ticks":1010000,"time":{"ms":5}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"9983abee-759d-478e-8c74-d6eb4898ca24","uptime":{"ms":3817770060}},"memstats":{"gc_next":20841712,"memory_alloc":10768704,"memory_total":127465004408},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.02,"15":0.05,"5":0.04,"norm":{"1":0.005,"15":0.0125,"5":0.01}}}}}}
Sep 8 13:54:31 bid-uat-kbn kibana: {"type":"log","@timestamp":"2021-09-08T06:54:31Z","tags":["error","plugins","taskManager","taskManager"],"pid":22453,"message":"Failed to poll for work: Error: work has timed out"}
elasticsearch log
Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36222}
[2021-09-08T13:52:14,323][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.18.238:50678}
[2021-09-08T13:52:14,326][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36224}
[2021-09-08T13:52:19,329][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36226}
[2021-09-08T13:52:21,341][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.102:55976}
[2021-09-08T13:52:24,333][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36228}
[2021-09-08T13:52:25,676][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [winlogbeat] for index [winlogbeat-7.10.2-2021.08.22-000005] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,677][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [.lists-default] for index [.lists-default-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [70]
[2021-09-08T13:52:25,677][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [metricbeat] for index [metricbeat-7.13.4-2021.09.02-000002] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,678][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [auditbeat] for index [auditbeat-7.11.2-2021.08.22-000005] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,678][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [kibana-event-log-policy] for index [.kibana-event-log-7.10.1-000005] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,678][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [ilm-history-ilm-policy] for index [ilm-history-3-000005] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,678][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [filebeat] for index [filebeat-7.10.1-2021.08.22-000005] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [69]
[2021-09-08T13:52:25,678][INFO ][o.e.x.i.IndexLifecycleRunner] [bdi-uat-els] policy [.items-default] for index [.items-default-000001] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [70]
[2021-09-08T13:52:26,645][WARN ][o.e.c.r.a.DiskThresholdMonitor] [bdi-uat-els] high disk watermark [90%] exceeded on [J6NuHKl8RieQhZib9BALmQ][bdi-uat-els][/data/elasticsearch/nodes/0] free: 8gb[8%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete
[2021-09-08T13:52:28,234][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.43:59452}
[2021-09-08T13:52:28,863][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.102:55978}
[2021-09-08T13:52:29,338][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36230}
[2021-09-08T13:52:30,756][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.43:59454}
[2021-09-08T13:52:34,341][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36232}
[2021-09-08T13:52:39,344][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36234}
[2021-09-08T13:52:44,347][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36236}
[2021-09-08T13:52:49,361][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36238}
[2021-09-08T13:52:54,365][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36240}
[2021-09-08T13:52:59,369][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36242}
[2021-09-08T13:53:04,371][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36244}
[2021-09-08T13:53:05,278][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.102:55980}
[2021-09-08T13:53:08,055][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.18.237:58502}
[2021-09-08T13:53:09,374][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36246}
[2021-09-08T13:53:09,883][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.18.238:50696}
[2021-09-08T13:53:14,378][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36248}
[2021-09-08T13:53:19,382][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.194.11.69:36250}
[2021-09-08T13:53:20,383][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [bdi-uat-els] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/10.194.11.67:9200, remoteAddress=/10.197.23.102:55982}
btw @warkolm can you help me?
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.