We have an elasticsearch cluster in docker swarm that consists of:
3 x controllers
1 x hot node
1 x warm node
1 x cold node
The cluster is showing as healthy and the nodes all have quite low memory/CPU usage however when trying to load logs it takes forever to load and when we get this error
Error: Batch request failed with status 502 at search_interceptor_SearchInterceptor.handleSearchError
I can see that indexing is happening between the nodes and wondering if that is overloading the cluster as I can see some timeout errors in Kibana, but surely we would see high CPU/mem is this was the case?
Is there anything I can look at to diagnose this issue? Thank you!