# Kibana site can't be reached

**URL:** <https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607>\
**Category:** Kibana\
**Created:** [March 24, 2022, 4:26pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607 "2022-03-24T16:26:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![spo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spo/32/78055_2.png) [@spo](https://discuss.elastic.co/u/spo)\
**Post date:** [March 24, 2022, 4:26pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/1 "2022-03-24T16:26:18Z")

</div>

I have not made any changes to the elk stack .It was working fine till yesterday, today at the beginning it did not let me login to the kibana portal.I cleared browser history restarted Elasticsearch and kibana and tried again this time kibana portal gives an error "THE SITE CAN'T BE REACHED" .

systemctl status Elasticsearch

```auto
elasticsearch.service - Elasticsearch
     Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-24 15:41:55 UTC; 25min ago
       Docs: https://www.elastic.co
   Main PID: 4930 (java)
      Tasks: 144 (limit: 19144)
     Memory: 10.4G
     CGroup: /system.slice/elasticsearch.service
             ├─4930 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.e>
             └─5124 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Mar 24 15:41:32 ip-xx-x-xx-xxx systemd[1]: Starting Elasticsearch...
Mar 24 15:41:55 ip-xx-x-xx-xxx systemd[1]: Started Elasticsearch.

```

systemctl status kibana

```auto
kibana.service - Kibana
     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-03-24 16:08:22 UTC; 18s ago
       Docs: https://www.elastic.co
   Main PID: 6004 (node)
      Tasks: 11 (limit: 19144)
     Memory: 438.4M
     CGroup: /system.slice/kibana.service
             └─6004 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist --logging.dest=/var/log/kibana/kibana.log --pid.file=/run/kibana/kibana.pid --deprecation.skip_deprecated_set>

Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: kibana.service: Scheduled restart job, restart counter is at 32.
Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: Stopped Kibana.
Mar 24 16:08:22 ip-xx-x-xx-xxx systemd[1]: Started Kibana.

```

I waited for a long time after restarting kibana and elsticsearch

journalctl -u kibana.service

```auto
Mar 24 15:51:27 ip-xx-x-xx-xxx kibana[5490]: FATAL Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpecte>
Mar 24 15:51:27 ip-xx-x-xx-xxx systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Mar 24 15:51:27 ip-xx-x-xx-xxx systemd[1]: kibana.service: Failed with result 'exit-code'.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: kibana.service: Scheduled restart job, restart counter is at 11.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: Stopped Kibana.
Mar 24 15:51:30 ip-xx-x-xx-xxx systemd[1]: Started Kibana.

```

cat Elasticsearch.yml

```auto
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.authc.api_key.enabled: true
cluster.name: "xyz"
node.name: abc
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: xx-x-xx-xxx 
discovery.type: single-node

```

cat kibana.yml

```auto
server.port: 5601
server.host: "xx-x-xx-xxx"
elasticsearch.hosts: ["http://xx-x-xx-xxx:9200"]
elasticsearch.username: "kibana_system"
elasticsearch.password: " ********"

```

---

<div class="post-metadata">

**Author:** ![bhavyarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavyarm/32/22392_2.png) [@bhavyarm](https://discuss.elastic.co/u/bhavyarm)\
**Post date:** [March 24, 2022, 4:53pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/2 "2022-03-24T16:53:22Z")

</div>

I see there are saved object migration errors in your kibana logs. Can you please check if there is more of them and details?

CC @LeeDr

Thanks,  
Bhavya

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [March 25, 2022, 5:06pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/3 "2022-03-25T17:06:18Z")

</div>

> [@spo](#):
>
> ```auto
> FATAL Error: Unable to complete saved object migrations for the [.kibana] index. Please check the health of your Elasticsearch cluster and try again. Unexpecte>
> 
> ```

Did you check the Elasticsearch logs? Maybe it ran out of disk space or something like that? If it hits the high watermark it would set indices to read-only and Kibana would fail to do it's startup tasks.

You could also use curl to check your Elasticsearch cluster health and post results back here. Things like;  
`curl http://localhost:9200`  
or if you have security enabled, you can use your password in place of "changeme" here;  
`curl http://elastic:changeme@localhost:9200`

And then also;  
`curl http://elastic:changeme@localhost:9200/_cat/health`

---

<div class="post-metadata">

**Author:** ![spo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spo/32/78055_2.png) [@spo](https://discuss.elastic.co/u/spo)\
**Post date:** [March 25, 2022, 6:11pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/4 "2022-03-25T18:11:53Z")

</div>

Thank You for your response. yes the issue was EBS volume was almost full , so once i increased the size the issue was resolved. but now i see something new.

docker logs 'name of the container' --tail 100 -f

```auto
2022-03-25T18:01:34,170][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@xx.x.xx.xxx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://xx.x.xx.xxx:9200/][Manticore::ConnectTimeout] Connect to xx.x.xx.xxx:9200 [/xx.x.xx.xxx]failed: connect timed out"}

```

i see the above error and the recent logs are not coming in to kibana  
restarted Elasticsearch , kibana and the docker containers for logstash couple of times but the error keeps repeating .though i did not change anything in the config files from earlier.it was working fine before and now i get this error.

output from Elasticsearch log file

```auto
2022-03-25T14:25:50,369][INFO][o.e.x.t.t.TransformPersistentTasksExecutor] [NAME OF THE ELK SERVER] [endpoint.metadata_current-default-1.2.2] successfully completed and scheduled task in node operation
[2022-03-25T14:25:50,645][INFO][o.e.x.t.t.TransformPersistentTasksExecutor] [NAME OF THE ELK SERVER] [endpoint.metadata_united-default-1.2.2] successfully completed and scheduled task in node operation
[2022-03-25T14:26:49,413][ERROR][o.e.x.d.l.DeprecationIndexingComponent] [NAME OF THE ELK SERVER] Bulk write of deprecation logs encountered some failures: [[JfF4wX8Ba2T_dwU6H1wx UnavailableShardsException[[.ds-.logs-deprecation.elasticsearch-default-2022.03.17-000006][0] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[.ds-.logs-deprecation.elasticsearch-default-2022.03.17-000006][0]] containing [index {[.logs-deprecation.elasticsearch-default][_doc][JfF4wX8Ba2T_dwU6H1wx], source[{"event.dataset": "deprecation.elasticsearch", "@timestamp": "2022-03-25T14:25:43,751Z", "log.level": "WARN", "log.logger": "org.elasticsearch.deprecation.common.settings.Settings", "elasticsearch.cluster.name": "elk.webappuat.com", "elasticsearch.cluster.uuid": "", "elasticsearch.node.id": "", "elasticsearch.node.name": "NAME OF THE ELK SERVER", "trace.id": "", "message": "[xpack.monitoring.collection.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.", "data_stream.type": "logs", "data_stream.dataset": "deprecation.elasticsearch", "data_stream.namespace": "default", "ecs.version": "1.7", "elasticsearch.event.category":"settings", "event.code": "xpack.monitoring.collection.enabled", "elasticsearch.http.request.x_opaque_id": "" }
]}]]]]]
[2022-03-25T14:26:52,836][INFO][o.e.c.r.a.AllocationService] [NAME OF THE ELK SERVER] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.transform-notifications-000002][0]]]).
[2022-03-25T14:28:44,289][INFO][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 11490 finished with response BulkByScrollResponse[took=39.2ms,timed_out=false,sliceId=null,updated=22,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T14:28:47,190][INFO][o.e.t.LoggingTaskListener] [ELK-IDC-NVIRGINIA-001] 11489 finished with response BulkByScrollResponse[took=2.9s,timed_out=false,sliceId=null,updated=6765,created=0,deleted=0,batches=7,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T14:28:51,011][INFO][o.e.x.i.a.TransportPutLifecycleAction] [NAME OF THE ELK SERVER] updating index lifecycle policy [.alerts-ilm-policy]
[2022-03-25T15:00:39,065][WARN][o.e.x.s.a.RealmsAuthenticator] [NAME OF THE ELK SERVER] Authentication to realm default_native failed - Password authentication failed for XYZ
[2022-03-25T15:17:14,371][INFO][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 164392 finished with response BulkByScrollResponse[took=97.7ms,timed_out=false,sliceId=null,updated=22,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T15:17:16,811][INFO][o.e.t.LoggingTaskListener] [NAME OF THE ELK SERVER] 164399 finished with response BulkByScrollResponse[took=2.5s,timed_out=false,sliceId=null,updated=6766,created=0,deleted=0,batches=7,versionConflicts=0,noops=0,retries=0,throttledUntil=0s,bulk_failures=[],search_failures=[]]
[2022-03-25T15:17:21,058][INFO][o.e.x.i.a.TransportPutLifecycleAction] [NAME OF THE ELK SERVER] updating index lifecycle policy [.alerts-ilm-policy]
"elk.xyz.com.log" 287L, 45676C 283,8 Bot

```

could this be anything related to the versions I am using  
Elasticsearch -7.17.1  
kibana- 7.17.1  
docker logstash -8.1.0

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [March 28, 2022, 1:53pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/5 "2022-03-28T13:53:17Z")

</div>

It could be a problem with those versions. We generally don't support anything connecting to Elasticsearch from a later version. Our upgrade instructions tell people to upgrade Elasticsearch first, then the other products. But I can't really say if that's the problem or not. I don't think anybody is testing the compatibility between those versions.

You could try just posting a doc to some new index and see if that works with the user/pwd that logstash is using.

---

<div class="post-metadata">

**Author:** ![spo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spo/32/78055_2.png) [@spo](https://discuss.elastic.co/u/spo)\
**Post date:** [March 28, 2022, 5:15pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/6 "2022-03-28T17:15:26Z")

</div>

Tried to post a doc to a new index .Unable to create new index.  
and in the docker logstash logs it gives the same warning

```auto
[2022-03-28T17:09:48,503][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@xx.x.xx.xxx:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://xx.x.xx.xxx:9200/][Manticore::ConnectTimeout] Connect to xx.x.xx.xxx:9200 [/xx.x.xx.xxx]failed: connect timed out"}

```

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [March 28, 2022, 8:03pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/7 "2022-03-28T20:03:13Z")

</div>

What error did you get trying to post a doc to a new index? Can you paste the response here?

What is \_cat/health showing now?

You might have to post a question in the Elasticsearch forum for help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 25, 2022, 8:04pm UTC](https://discuss.elastic.co/t/kibana-site-cant-be-reached/300607/8 "2022-04-25T20:04:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.