How to reduce thread-pool data rejection in elasticsearch cluster?

[varun1992 (Varun)]

please help me to find out network.host settings

what should be the value of the network.host ?

should i give public ip value or private ip value ?

i am having a confution in this settings. currently i gave

network.bind_host => private ip value
network.publish_host => public ip value

is this ok

[Christian_Dahlqvist]

What does that have to do with data rejection? Please open new questions in separate threads.

[varun1992 (Varun)]

i configured logstash in cluster. everything is green . but sometimes i am getting

**failed to send ping to #zen_unicast error**

why is that ? when i check manually through browser all nodes were responding fine

[jimyoo]

I tested some heavy injections with tuning and those under helped. In my test case, injection rate was higher than yours and used high-end servers. You may need to find your own numbers by testing as Christian suggested.

• more core, memory and using SSD.
• increased refresh interval; I used 60 sec.
• maintained index size: < 30GB per shard
• tuned index.translog.flush_threshold_size: I used 10G
• tuned bulk size (buffering before sending out to Elasticsearch): ~30MB
• To get these numbers: I pushed injection rate higher until I got rejections (429, Too many requests)

I didn’t use beats but it seems you can tune bulk size (buffering) in beats. For example,
https://www.elastic.co/guide/en/beats/packetbeat/6.2/configuring-internal-queue.html