# Issue with bulk index performance **URL:** \ **Category:** Elasticsearch\ **Created:** [June 4, 2019, 9:05pm UTC](https://discuss.elastic.co/t/issue-with-bulk-index-performance/184253 "2019-06-04T21:05:13Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![opak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/opak/32/5705_2.png) [@opak](https://discuss.elastic.co/u/opak)\ **Post date:** [June 4, 2019, 9:05pm UTC](https://discuss.elastic.co/t/issue-with-bulk-index-performance/184253/1 "2019-06-04T21:05:13Z")
Hello, we have issue with our ES cluster 6.6.2 while we try to index 1m documents. our index: ```auto {"settings":{"index":{"number_of_shards":"5","provided_name":"org220_dev_1559679865_elixir","creation_date":"1559679865761","analysis":{"filter":{"instant_filter":{"type":"edge_ngram","min_gram":"1","max_gram":"20"},"similarity_stop":{"ignore_case":"true","type":"stop","stopwords":["a","about","above","after","again","against","ain","all","am","an","and","any","are","arent","as","at","be","because","been","before","being","below","between","both","but","by","can","couldnt","did","didn","didnt","do","does","doesn","doesnt","doing","don","dont","down","during","each","few","for","from","further","had","hadn","hadnt","has","hasnt","have","havent","having","he","her","here","hers","herself","him","himself","his","how","i","if","in","into","is","isnt","it","its","its","itself","just","ll","m","ma","me","mightnt","more","most","mustnt","my","myself","neednt","no","nor","not","now","o","of","off","on","once","only","or","other","our","ours","ourselves","out","over","own","re","s","same","shant","she","shes","should","shouldve","shouldnt","so","some","such","t","than","that","thatll","the","their","theirs","them","themselves","then","there","these","they","this","those","through","to","too","under","until","up","ve","very","was","wasnt","we","were","werent","what","when","where","which","while","who","whom","why","will","with","wont","wouldnt","y","you","youd","youll","youre","youve","your","yours","yourself","yourselves",""]}},"analyzer":{"whitespace_lowercase":{"filter":["lowercase"],"type":"custom","tokenizer":"whitespace"},"fulltext_analyzer":{"filter":["standard","lowercase","similarity_stop","type_as_payload"],"char_filter":["html_strip","similarity_char_filter"],"type":"custom","tokenizer":"whitespace"},"instantsearch":{"filter":["lowercase","instant_filter"],"type":"custom","tokenizer":"whitespace"}},"char_filter":{"similarity_char_filter":{"pattern":"[^a-zA-Z0-9]","type":"pattern_replace","replacement":""}}},"number_of_replicas":"2","uuid":"jrp4t8njTfyz0N4ekMwP-g","version":{"created":"6060299"}}}} ``` mappings: ```auto {"index_name":{"mappings":{"_doc":{"dynamic":"true","dynamic_templates":[{"segments_strings":{"path_match":"segments.*","path_unmatch":"segments.*.value","match_mapping_type":"string","mapping":{"index":true,"type":"keyword"}}},{"segments_value":{"path_match":"segments.*.value","mapping":{"index":true,"type":"keyword"}}},{"user_meta_value":{"path_match":"user_meta.*.value","mapping":{"index":true,"type":"text"}}},{"user_metas":{"path_match":"user_meta.*","path_unmatch":"user_meta.*.value","match_mapping_type":"string","mapping":{"index":true,"type":"keyword"}}},{"tags":{"path_match":"tenants.*.tags","mapping":{"index":true,"type":"keyword"}}},{"tag_ids":{"path_match":"tenants.*.tag_ids","mapping":{"index":true,"type":"keyword"}}}],"properties":{"added_at":{"type":"date"},"approved":{"type":"boolean"},"approved_at":{"type":"date"},"comment":{"type":"text","analyzer":"instantsearch","search_analyzer":"whitespace_lowercase"},"created_at":{"type":"date"},"data_source":{"properties":{"id":{"type":"integer"},"name":{"type":"keyword"}}},"data_type":{"properties":{"id":{"type":"integer"},"name":{"type":"keyword"}}},"dataset":{"properties":{"id":{"type":"integer"},"key":{"type":"keyword"},"name":{"type":"keyword"}}},"datatype":{"properties":{"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"has_comment":{"type":"boolean"},"has_themes":{"type":"boolean"},"hidden":{"type":"boolean"},"locker_id":{"type":"integer"},"marked_for_supervisor":{"type":"boolean"},"marvin_approved":{"type":"boolean"},"marvin_processed":{"type":"boolean"},"organization_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"original_comment":{"type":"text","analyzer":"instantsearch","search_analyzer":"whitespace_lowercase"},"score":{"type":"integer"},"segments":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"supervisor_approved":{"type":"boolean"},"tenants":{"properties":{"tenant_220":{"properties":{"tag_ids":{"type":"keyword"},"tags":{"type":"keyword"}}}}},"term_vector_comment":{"type":"text","store":true,"term_vector":"with_positions_offsets_payloads","analyzer":"fulltext_analyzer"},"themes":{"type":"nested","properties":{"name":{"type":"keyword"},"parent":{"type":"keyword"},"sentiment":{"type":"integer"}}},"themes_verified":{"type":"boolean"},"type":{"type":"keyword"},"updated_at":{"type":"date"},"user_meta":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}} ``` We have 1m +- docs into our database, and we want to have higt performance to index this docs. We build cluster with 4 data nodes (4g 4g), 2 masters (2g 2g), and 2 clients (1g 1g), all this we are running into kubernetes. Then we create an erlang client, which gives us full power for parallel requests. All looks good, but after 890k, we start to see, that new docs / requests does't create new documents. But response from bulk requests without any errors. But when some another documents come to another indexes (we have \> 400 indexes) we have this error: ```auto [429] {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[es-data-300-3][10.244.12.220:9300][indices:data/write/bulk[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [31516653][indices:data/write/bulk[s][p]]: request: BulkShardRequest ``` So my question, how we can fix this error, and why we don't have 419 for bulk request? It's looks like a bug, because if we have 419, we can retry like logstash or apm products. Looks like a bug for 6.6.2. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 2, 2019, 9:10pm UTC](https://discuss.elastic.co/t/issue-with-bulk-index-performance/184253/2 "2019-07-02T21:10:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.