# Curator not working with istio - fail to reach ES sporadically

**URL:** <https://discuss.elastic.co/t/curator-not-working-with-istio-fail-to-reach-es-sporadically/234721>\
**Category:** Elasticsearch\
**Tags:** curator\
**Created:** [May 28, 2020, 10:41am UTC](https://discuss.elastic.co/t/curator-not-working-with-istio-fail-to-reach-es-sporadically/234721 "2020-05-28T10:41:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivani\_aggarwal](https://avatars.discourse-cdn.com/v4/letter/s/22d042/32.png) [@shivani\_aggarwal](https://discuss.elastic.co/u/shivani_aggarwal)\
**Post date:** [May 28, 2020, 10:41am UTC](https://discuss.elastic.co/t/curator-not-working-with-istio-fail-to-reach-es-sporadically/234721/1 "2020-05-28T10:41:33Z")

</div>

Hi,  
I am trying to run elasticsearch with istio in kubernetes environment.  
**Elasticsearch-oss: 7.0.1**  
**Istio 1.5.2**  
Istio sidecars are ingested into elasticsearch pods (master, data nodes) as well as into elasticsearch-curator pod.

The elasticsearch cluster formation is happening fine, the elasticsearch REST api keeps responding.  
Curator is configured to run as a k8s cronjob every 5 min to delete old indices. We observe at times, curator is able to reach ES and delete indices as expected - resulting in jobs to succeed.  
But a lot of times, it fails to reach ES with this error msg -

```
2020-05-27 01:01:36,036 ERROR HTTP N/A error: HTTPConnectionPool(host='elasticsearch', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0c41220dd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
2020-05-27 01:01:36,037 CRITICAL Curator cannot proceed. Exiting.

```

I triggered a script that continuously contacts Elasticsearch REST api on 9200 (\_cat/nodes) every 5 seconds. The service was responsive & reachable all the time.

The same chart & configurations when installed without istio enabled in the namespace, work fine. Can you help us find the root cause of the issue? Are any additional configurations required?

Curator with debug logs attached (failed case) - [https://gist.github.com/aggarwalShivani/c137c0baecf047087f30116d70959f61](https://gist.github.com/aggarwalShivani/c137c0baecf047087f30116d70959f61)

Curator logs with istio (success case) - [https://gist.github.com/aggarwalShivani/c24c658066b1c8357ef2b2fdfa0b9118](https://gist.github.com/aggarwalShivani/c24c658066b1c8357ef2b2fdfa0b9118)

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [May 28, 2020, 1:50pm UTC](https://discuss.elastic.co/t/curator-not-working-with-istio-fail-to-reach-es-sporadically/234721/2 "2020-05-28T13:50:58Z")

</div>

I cannot begin to explain why you would see this error message, but it is not a Curator problem.

> `ConnectionRefusedError: [Errno 111] Connection refused`

Curator isn't the source of this message, but `urllib3`, the underlying HTTP connectivity module. Curator is unable to connect not because the target remote is unavailable, but that it is _refusing_ to permit Curator (well, urllib3) to connect. So, it seems to me that whatever is managing connections in Kubernetes is a good place to start looking. Refused could mean access control is denying access, for example. I also cannot explain why it works occasionally, and not always. It's something inside k8s, is my guess.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 25, 2020, 1:51pm UTC](https://discuss.elastic.co/t/curator-not-working-with-istio-fail-to-reach-es-sporadically/234721/3 "2020-06-25T13:51:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.