I am trying to run elasticsearch with istio in kubernetes environment.
Istio sidecars are ingested into elasticsearch pods (master, data nodes) as well as into elasticsearch-curator pod.
The elasticsearch cluster formation is happening fine, the elasticsearch REST api keeps responding.
Curator is configured to run as a k8s cronjob every 5 min to delete old indices. We observe at times, curator is able to reach ES and delete indices as expected - resulting in jobs to succeed.
But a lot of times, it fails to reach ES with this error msg -
2020-05-27 01:01:36,036 ERROR HTTP N/A error: HTTPConnectionPool(host='elasticsearch', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0c41220dd0>: Failed to establish a new connection: [Errno 111] Connection refused')) 2020-05-27 01:01:36,037 CRITICAL Curator cannot proceed. Exiting.
I triggered a script that continuously contacts Elasticsearch REST api on 9200 (_cat/nodes) every 5 seconds. The service was responsive & reachable all the time.
The same chart & configurations when installed without istio enabled in the namespace, work fine. Can you help us find the root cause of the issue? Are any additional configurations required?
Curator with debug logs attached (failed case) - https://gist.github.com/aggarwalShivani/c137c0baecf047087f30116d70959f61
Curator logs with istio (success case) - https://gist.github.com/aggarwalShivani/c24c658066b1c8357ef2b2fdfa0b9118