# Elasticsearch Increasing Write rejections

**URL:** <https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882>\
**Category:** Elasticsearch\
**Created:** [August 18, 2021, 9:56pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882 "2021-08-18T21:56:08Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 18, 2021, 9:56pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/1 "2021-08-18T21:56:08Z")

</div>

Hello,  
I have 20 node platinum licenced ELK cluster. (Version 7.6.1)

Setup is stated as below.

16 data nodes --\> each 64GB memory // 32gb allocated as Xms-Xmx jvm . --\> 8TB disk each  
3 master nodes --\> 32gb memory each  
1 Ml node --\> 64 gb memory

 ![Nodes](https://us1.discourse-cdn.com/elastic/original/3X/d/e/de39c31fc22d9b8b72d246d6934a4d8c48c728e0.jpeg)

My problem is i checked most of the index tuning advisory and threadpool write rejection cases somehow my rejections started to increase for a while , 2 nodes nearly have %5 rejection rate.

![threadpool](https://us1.discourse-cdn.com/elastic/original/3X/f/9/f90b65c1b15abcd15efe8411a1862f9052bbcd55.jpeg)

My indexing rates are like as stated below , the spikes are huge but i get data from Database, syslog, log files and directly forwarded to my elasticsearch from fluentbit or logstash of some other services all sources have different behaviour.

 ![Indexing](https://us1.discourse-cdn.com/elastic/original/3X/9/b/9bc67985f8ee3707782fbe95ae29425c9fe0cb2f.jpeg)

My most used index template is as below. So most of the indexes are 4 primary and 1 replica.  
Refresh interval may be better if higher value i guess.

```auto
> {
> "main_temp" : {
> "order" : 0,
> "index_patterns" : [
> "csm_siem_record_*",
> "csm_siem_restored_*"
> ],
> "settings" : {
> "index" : {
> "codec" : "best_compression",
> "refresh_interval" : "5s",
> "analysis" : {
> "filter" : {
> "ntk_asciifolding" : {
> "type" : "asciifolding",
> "preserve_original" : "false"
> },
> "ntk_turkce_lowercase" : {
> "type" : "lowercase",
> "language" : "turkish"
> }
> },
> "analyzer" : {
> "raw_log" : {
> "filter" : [
> "ntk_turkce_lowercase",
> "ntk_asciifolding"
> ],
> "type" : "custom",
> "tokenizer" : "classic"
> }
> }
> },
> "number_of_shards" : "4",
> "auto_expand_replicas" : "0-1",
> "query" : {
> "default_field" : "raw_record"
> }
> }
> },
> "mappings" : {
> "dynamic_templates" : [
> {
> "notanalyzed" : {
> "mapping" : {
> "type" : "keyword"
> },
> "match_mapping_type" : "string",
> "match" : "*"
> }
> }
> ],
> "date_detection" : false,
> "properties" : {
> "received_bytes" : {
> "type" : "long"
> },
> "destination_port" : {
> "type" : "integer"
> },
> "jvbrtt" : {
> "type" : "short"
> },
> "sent_packets" : {
> "type" : "long"
> },
> "responseData" : {
> "ignore_above" : 8191,
> "type" : "keyword"
> },
> "translated_source_ip" : {
> "type" : "ip"
> },
> "raw_record" : {
> "analyzer" : "raw_log",
> "type" : "text"
> },
> "packets" : {
> "type" : "long"
> },
> "session_time" : {
> "type" : "long"
> },
> "source_ip" : {
> "type" : "ip"
> },
> "sent_bytes" : {
> "type" : "long"
> },
> "download" : {
> "type" : "short"
> },
> "duration" : {
> "type" : "long"
> },
> "destination_ip" : {
> "type" : "ip"
> },
> "translated_destination_ip" : {
> "type" : "ip"
> },
> "translated_source_port" : {
> "type" : "integer"
> },
> "date_time" : {
> "format" : "strict_date_optional_time||epoch_millis",
> "type" : "date"
> },
> "translated_destination_port" : {
> "type" : "integer"
> },
> "source_port" : {
> "type" : "integer"
> },
> "connectionquality" : {
> "type" : "float"
> },
> "responsetime" : {
> "type" : "keyword"
> },
> "issuccessful" : {
> "type" : "keyword"
> },
> "requestData" : {
> "ignore_above" : 8191,
> "type" : "keyword"
> },
> "received_packets" : {
> "type" : "long"
> },
> "severity" : {
> "type" : "integer"
> },
> "responsecode" : {
> "type" : "keyword"
> },
> "coordinate" : {
> "type" : "geo_point"
> },
> "destination_ip_coordinate" : {
> "type" : "geo_point"
> },
> "count" : {
> "type" : "long"
> },
> "maxframeheight" : {
> "type" : "integer"
> },
> "translated_ip" : {
> "type" : "ip"
> },
> "upload" : {
> "type" : "short"
> },
> "resp_time" : {
> "type" : "integer"
> },
> "message" : {
> "analyzer" : "raw_log",
> "type" : "text"
> },
> "isservererror" : {
> "type" : "keyword"
> },
> "sign_time" : {
> "format" : "strict_date_optional_time||epoch_millis",
> "type" : "date"
> },
> "device_ip" : {
> "type" : "ip"
> },
> "source_ip_coordinate" : {
> "type" : "geo_point"
> },
> "upload__1" : {
> "type" : "short"
> },
> "download__1" : {
> "type" : "short"
> },
> "bytes" : {
> "type" : "long"
> },
> "starttime" : {
> "type" : "keyword"
> },
> "lastn" : {
> "type" : "short"
> },
> "email_subject" : {
> "analyzer" : "raw_log",
> "type" : "text"
> }
> }
> },
> "aliases" : { }
> }
> }

```

I will be really glad if you have further recommendations.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 18, 2021, 9:59pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/2 "2021-08-18T21:59:08Z")

</div>

Hi @Tombal

As a platinum customer you are entitled to support on these kind of issues.

Have you filed a support ticket, Elastic support is not just break / fix they certainly can help take a look at this with you and help diagnose and make suggestions.

That would be my first suggestion.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 18, 2021, 10:01pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/3 "2021-08-18T22:01:10Z")

</div>

Also my quick notice is that you say that you set the JVM to 32 GB,  
That is actually not optimum the JVM needs to be set somewhere between 26 to 30 GB otherwise you are no longer using compressed object pointers and will actually make the system less performant.

See here

> **[Setting the heap size | Elasticsearch Guide \[7.6\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/heap-size.html)**

I'm not saying that's a source your problem but it's certainly not optimal.

---

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 18, 2021, 10:06pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/4 "2021-08-18T22:06:48Z")

</div>

Thx Stephen , i will have a look.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 19, 2021, 5:19am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/5 "2021-08-19T05:19:01Z")

</div>

There have been [improvements in this area in recent versions](https://discuss.elastic.co/t/reduce-bulk-rejections/281782/4) so it might be a good idea to upgrade.

---

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 19, 2021, 6:57am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/6 "2021-08-19T06:57:16Z")

</div>

This is also on my schedule... Need to make sure every job working atm is completely suitable with newer versions.

By the way , made some adjustments on the worst situation nodes and after final restart and adding index\_buffer\_size parameter after 4 hours cluster got green. Now the rejection rate is %0.01 and even lower than 1 of 10000 . That is becoming more acceptable.

The highest queue size doesn't make it any better after 500 for me , so i lowered it to 400.  
Buffer size parameter was not added so was default to %10 i guess.

> thread\_pool.write.queue\_size: 400  
> indices.memory.index\_buffer\_size: 40%

I hope it works for some other people...

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 19, 2021, 7:02am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/7 "2021-08-19T07:02:01Z")

</div>

> [@Tombal](#):
>
> thread\_pool.write.queue\_size: 400

That's not really a good idea. Take a look at [Any idea what these errors mean version 2.4.2 - #5 by jasontedor](https://discuss.elastic.co/t/any-idea-what-these-errors-mean-version-2-4-2/70690/5) for a good overview of why

---

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 19, 2021, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/8 "2021-08-19T07:51:10Z")

</div>

You are right , the higher queue makes it worse . It doesn't fix the problem. It is still out there.  
So i lowered it , maybe i'll remove the setting as well , trying every possible combination.

My other data nodes doesn't have this setting in elasticsearch.yml .  
I would be glad to take further advice about any other handy settings.

Thx Warkolm...

---

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 21, 2021, 8:47am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/9 "2021-08-21T08:47:37Z")

</div>

Just to give brief info in case it helps others;

My final numbers atm;

```auto

thread_pool.write.queue_size: 250
indices.memory.index_buffer_size: 30%

```

(higher percentages like 40,50 seemed to lag servers a bit and didn't make much difference for me - kibana started to lose connection with nodes not sure if it is about it but \_cat/nodes started to take over 1.5min)

İndex template and active index changes;

increased `"index.refresh.interval"` from 5s to 15s , some of my data has to be searched close to real time.

added

`"index.translog.durability" : "async"`

(default is indexing per request-event , now it is by default for every 5seconds if i'm not mistaken , will watch for a while may change it to 10secs later)

Also changed 1 index template for daily indexes which doesn't keep huge data.  
My default was `"number of shards": 4 , "auto_expand_replicas" : "0-1"` , on these lowered it to `"number_of_shards": 3.`

My retention period changes according to indexes.  
Either 15 days,2 months or up to 2 years (month based indexes). Lowered shard count for close to 2 years indices with SHRINK api.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 21, 2021, 2:51pm UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/10 "2021-08-21T14:51:14Z")

</div>

@Tombal

Thanks for sharing your updates

Perhaps, another setting you can look at is this, this helped on a heavy write use case I had increasing the `flush_threshold_size`

My settings on this particular index, windows event logs

```auto
    },
    "refresh_interval": "30s",
    "number_of_shards": "4",
    "translog": {
      "flush_threshold_size": "2gb"
    },

```

---

<div class="post-metadata">

**Author:** ![Tombal](https://avatars.discourse-cdn.com/v4/letter/t/977dab/32.png) [@Tombal](https://discuss.elastic.co/u/Tombal)\
**Post date:** [August 22, 2021, 2:18am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/11 "2021-08-22T02:18:51Z")

</div>

Thx Steph...  
I'll slightly increase refresh\_interval and flush\_threshold\_size to see difference step by step like (20sec and 1gb for start)

My current rates atm ;  
there is only one node which has a rejection rate of %0.2 other nodes either ok or like 1/100000 or so this reject node has some heavy indexed indice shards. So i guess i'm nearly done with my problems. Some slight improvements will make it ok.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2021, 2:18am UTC](https://discuss.elastic.co/t/elasticsearch-increasing-write-rejections/281882/12 "2021-09-19T02:18:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.