How to reduce thread-pool data rejection in elasticsearch cluster?

Elastic Stack Elasticsearch
21

please help me to find out network.host settings

what should be the value of the network.host ?

should i give public ip value or private ip value ?

i am having a confution in this settings. currently i gave

network.bind_host => private ip value
network.publish_host => public ip value

is this ok

22

What does that have to do with data rejection? Please open new questions in separate threads.

23

i configured logstash in cluster. everything is green . but sometimes i am getting

**failed to send ping to #zen_unicast error**

why is that ? when i check manually through browser all nodes were responding fine

24

I tested some heavy injections with tuning and those under helped. In my test case, injection rate was higher than yours and used high-end servers. You may need to find your own numbers by testing as Christian suggested.

  • more core, memory and using SSD.
  • increased refresh interval; I used 60 sec.
  • maintained index size: < 30GB per shard
  • tuned index.translog.flush_threshold_size: I used 10G
  • tuned bulk size (buffering before sending out to Elasticsearch): ~30MB
  • To get these numbers: I pushed injection rate higher until I got rejections (429, Too many requests)

I didn’t use beats but it seems you can tune bulk size (buffering) in beats. For example,
https://www.elastic.co/guide/en/beats/packetbeat/6.2/configuring-internal-queue.html

26

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.