Hi , I am running a 3 node elastic search cluster with replica 1 . I am also running two node logstash HA. Problem is that recently i have seen this error "courier fetch : X of X shards failed " . But if i look at my cluster heath it is green and shards are all in started mode. No problem with the cluster whatsoever.
Even on kibana if i look at the dashboards that i have created earlier , they all showing logs. logstash is parsing logs correctly , but kibana still show the problem . i tried to use this configuration and tried rolling restart . Nothing seem helps.
threadpool.search.type: fixed
threadpool.search.size: 200
threadpool.search.queue_size: 10000
I also checked how many threads elasticsearch uses and found in all node , total number of threads are pretty much less than 100. Any idea what is causing it ?
Any help is very much appreciated.