Kibana site can't be reached

spo · March 24, 2022, 4:26pm

I have not made any changes to the elk stack .It was working fine till yesterday, today at the beginning it did not let me login to the kibana portal.I cleared browser history restarted Elasticsearch and kibana and tried again this time kibana portal gives an error "THE SITE CAN'T BE REACHED" .

systemctl status Elasticsearch

elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-24 15:41:55 UTC; 25min ago
       Docs: https://www.elastic.co
   Main PID: 4930 (java)
      Tasks: 144 (limit: 19144)
     Memory: 10.4G
     CGroup: /system.slice/elasticsearch.service
             ├─4930 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.e>
             └─5124 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Mar 24 15:41:32 ip-xx-x-xx-xxx systemd[1]: Starting Elasticsearch...
Mar 24 15:41:55 ip-xx-x-xx-xxx systemd[1]: Started Elasticsearch.

systemctl status kibana

kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-24 16:08:22 UTC; 18s ago
       Docs: https://www.elastic.co
   Main PID: 6004 (node)
      Tasks: 11 (limit: 19144)
     Memory: 438.4M
     CGroup: /system.slice/kibana.service
             └─6004 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid --deprecation.skip_deprecated_set>

Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: kibana.service: Scheduled restart job, restart counter is at 32.
Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: Stopped Kibana.
Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: Started Kibana.

I waited for a long time after restarting kibana and elsticsearch

journalctl -u kibana.service

Mar 24 15:51:27 ip-xx-x-xx-xxx kibana[5490]:  FATAL  Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpecte>
Mar 24 15:51:27 ip-xx-x-xx-xxx systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Mar 24 15:51:27 ip-xx-x-xx-xxx systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: kibana.service: Scheduled restart job, restart counter is at 11.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: Stopped Kibana.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: Started Kibana.

cat Elasticsearch.yml

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.authc.api_key.enabled: true
cluster.name: "xyz"
node.name: abc
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: xx-x-xx-xxx 
discovery.type: single-node

cat kibana.yml

server.port: 5601
server.host: "xx-x-xx-xxx"
elasticsearch.hosts: ["http://xx-x-xx-xxx:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: "********"

bhavyarm · March 24, 2022, 4:53pm

I see there are saved object migration errors in your kibana logs. Can you please check if there is more of them and details?

CC @LeeDr

Thanks,
Bhavya

LeeDr · March 25, 2022, 5:06pm

Did you check the Elasticsearch logs? Maybe it ran out of disk space or something like that? If it hits the high watermark it would set indices to read-only and Kibana would fail to do it's startup tasks.

You could also use curl to check your Elasticsearch cluster health and post results back here. Things like;
curl http://localhost:9200
or if you have security enabled, you can use your password in place of "changeme" here;
curl http://elastic:changeme@localhost:9200

And then also;
curl http://elastic:changeme@localhost:9200/_cat/health

spo · March 25, 2022, 6:11pm

Thank You for your response. yes the issue was EBS volume was almost full , so once i increased the size the issue was resolved. but now i see something new.

docker logs 'name of the container' --tail 100 -f

2022-03-25T18:01:34,170][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@xx.x.xx.xxx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://xx.x.xx.xxx:9200/][Manticore::ConnectTimeout] Connect to xx.x.xx.xxx:9200 [/xx.x.xx.xxx]failed: connect timed out"}

i see the above error and the recent logs are not coming in to kibana
restarted Elasticsearch , kibana and the docker containers for logstash couple of times but the error keeps repeating .though i did not change anything in the config files from earlier.it was working fine before and now i get this error.

output from Elasticsearch log file

2022-03-25T14:25:50,369][INFO ][o.e.x.t.t.TransformPersistentTasksExecutor] [NAME OF THE ELK SERVER] [endpoint.metadata_current-default-1.2.2] successfully completed and scheduled task in node operation
[2022-03-25T14:25:50,645][INFO ][o.e.x.t.t.TransformPersistentTasksExecutor] [NAME OF THE ELK SERVER] [endpoint.metadata_united-default-1.2.2] successfully completed and scheduled task in node operation
[2022-03-25T14:26:49,413][ERROR][o.e.x.d.l.DeprecationIndexingComponent] [NAME OF THE ELK SERVER] Bulk write of deprecation logs encountered some failures: [[JfF4wX8Ba2T_dwU6H1wx UnavailableShardsException[[.ds-.logs-deprecation.elasticsearch-default-2022.03.17-000006][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-.logs-deprecation.elasticsearch-default-2022.03.17-000006][0]] containing [index {[.logs-deprecation.elasticsearch-default][_doc][JfF4wX8Ba2T_dwU6H1wx], source[{"event.dataset": "deprecation.elasticsearch", "@timestamp": "2022-03-25T14:25:43,751Z", "log.level": "WARN", "log.logger": "org.elasticsearch.deprecation.common.settings.Settings", "elasticsearch.cluster.name": "elk.webappuat.com", "elasticsearch.cluster.uuid": "", "elasticsearch.node.id": "", "elasticsearch.node.name": "NAME OF THE ELK SERVER", "trace.id": "", "message": "[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "data_stream.type": "logs", "data_stream.dataset": "deprecation.elasticsearch", "data_stream.namespace": "default", "ecs.version": "1.7", "elasticsearch.event.category":"settings", "event.code": "xpack.monitoring.collection.enabled", "elasticsearch.http.request.x_opaque_id": "" }
]}]]]]]
[2022-03-25T14:26:52,836][INFO ][o.e.c.r.a.AllocationService] [NAME OF THE ELK SERVER] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.transform-notifications-000002][0]]]).
[2022-03-25T14:28:44,289][INFO ][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 11490 finished with response BulkByScrollResponse[took=39.2ms,timed_out=false,sliceId=null,updated=22,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T14:28:47,190][INFO ][o.e.t.LoggingTaskListener] [ELK-IDC-NVIRGINIA-001] 11489 finished with response BulkByScrollResponse[took=2.9s,timed_out=false,sliceId=null,updated=6765,created=0,deleted=0,batches=7,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T14:28:51,011][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [NAME OF THE ELK SERVER] updating index lifecycle policy [.alerts-ilm-policy]
[2022-03-25T15:00:39,065][WARN ][o.e.x.s.a.RealmsAuthenticator] [NAME OF THE ELK SERVER] Authentication to realm default_native failed - Password authentication failed for XYZ
[2022-03-25T15:17:14,371][INFO ][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 164392 finished with response BulkByScrollResponse[took=97.7ms,timed_out=false,sliceId=null,updated=22,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T15:17:16,811][INFO ][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 164399 finished with response BulkByScrollResponse[took=2.5s,timed_out=false,sliceId=null,updated=6766,created=0,deleted=0,batches=7,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T15:17:21,058][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [NAME OF THE ELK SERVER] updating index lifecycle policy [.alerts-ilm-policy]
"elk.xyz.com.log" 287L, 45676C                                                                                                                                                         283,8         Bot

could this be anything related to the versions I am using
Elasticsearch -7.17.1
kibana- 7.17.1
docker logstash -8.1.0

LeeDr · March 28, 2022, 1:53pm

It could be a problem with those versions. We generally don't support anything connecting to Elasticsearch from a later version. Our upgrade instructions tell people to upgrade Elasticsearch first, then the other products. But I can't really say if that's the problem or not. I don't think anybody is testing the compatibility between those versions.

You could try just posting a doc to some new index and see if that works with the user/pwd that logstash is using.

spo · March 28, 2022, 5:15pm

Tried to post a doc to a new index .Unable to create new index.
and in the docker logstash logs it gives the same warning

[2022-03-28T17:09:48,503][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@xx.x.xx.xxx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://xx.x.xx.xxx:9200/][Manticore::ConnectTimeout] Connect to xx.x.xx.xxx:9200 [/xx.x.xx.xxx]failed: connect timed out"}

LeeDr · March 28, 2022, 8:03pm

What error did you get trying to post a doc to a new index? Can you paste the response here?

What is _cat/health showing now?

You might have to post a question in the Elasticsearch forum for help.

system · April 25, 2022, 8:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.