Hello, we have issue with our ES cluster 6.6.2 while we try to index 1m documents.
our index:
{"settings":{"index":{"number_of_shards":"5","provided_name":"org220_dev_1559679865_elixir","creation_date":"1559679865761","analysis":{"filter":{"instant_filter":{"type":"edge_ngram","min_gram":"1","max_gram":"20"},"similarity_stop":{"ignore_case":"true","type":"stop","stopwords":["a","about","above","after","again","against","ain","all","am","an","and","any","are","arent","as","at","be","because","been","before","being","below","between","both","but","by","can","couldnt","did","didn","didnt","do","does","doesn","doesnt","doing","don","dont","down","during","each","few","for","from","further","had","hadn","hadnt","has","hasnt","have","havent","having","he","her","here","hers","herself","him","himself","his","how","i","if","in","into","is","isnt","it","its","its","itself","just","ll","m","ma","me","mightnt","more","most","mustnt","my","myself","neednt","no","nor","not","now","o","of","off","on","once","only","or","other","our","ours","ourselves","out","over","own","re","s","same","shant","she","shes","should","shouldve","shouldnt","so","some","such","t","than","that","thatll","the","their","theirs","them","themselves","then","there","these","they","this","those","through","to","too","under","until","up","ve","very","was","wasnt","we","were","werent","what","when","where","which","while","who","whom","why","will","with","wont","wouldnt","y","you","youd","youll","youre","youve","your","yours","yourself","yourselves",""]}},"analyzer":{"whitespace_lowercase":{"filter":["lowercase"],"type":"custom","tokenizer":"whitespace"},"fulltext_analyzer":{"filter":["standard","lowercase","similarity_stop","type_as_payload"],"char_filter":["html_strip","similarity_char_filter"],"type":"custom","tokenizer":"whitespace"},"instantsearch":{"filter":["lowercase","instant_filter"],"type":"custom","tokenizer":"whitespace"}},"char_filter":{"similarity_char_filter":{"pattern":"[^a-zA-Z0-9 ]","type":"pattern_replace","replacement":""}}},"number_of_replicas":"2","uuid":"jrp4t8njTfyz0N4ekMwP-g","version":{"created":"6060299"}}}}
mappings:
{"index_name":{"mappings":{"_doc":{"dynamic":"true","dynamic_templates":[{"segments_strings":{"path_match":"segments.*","path_unmatch":"segments.*.value","match_mapping_type":"string","mapping":{"index":true,"type":"keyword"}}},{"segments_value":{"path_match":"segments.*.value","mapping":{"index":true,"type":"keyword"}}},{"user_meta_value":{"path_match":"user_meta.*.value","mapping":{"index":true,"type":"text"}}},{"user_metas":{"path_match":"user_meta.*","path_unmatch":"user_meta.*.value","match_mapping_type":"string","mapping":{"index":true,"type":"keyword"}}},{"tags":{"path_match":"tenants.*.tags","mapping":{"index":true,"type":"keyword"}}},{"tag_ids":{"path_match":"tenants.*.tag_ids","mapping":{"index":true,"type":"keyword"}}}],"properties":{"added_at":{"type":"date"},"approved":{"type":"boolean"},"approved_at":{"type":"date"},"comment":{"type":"text","analyzer":"instantsearch","search_analyzer":"whitespace_lowercase"},"created_at":{"type":"date"},"data_source":{"properties":{"id":{"type":"integer"},"name":{"type":"keyword"}}},"data_type":{"properties":{"id":{"type":"integer"},"name":{"type":"keyword"}}},"dataset":{"properties":{"id":{"type":"integer"},"key":{"type":"keyword"},"name":{"type":"keyword"}}},"datatype":{"properties":{"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"has_comment":{"type":"boolean"},"has_themes":{"type":"boolean"},"hidden":{"type":"boolean"},"locker_id":{"type":"integer"},"marked_for_supervisor":{"type":"boolean"},"marvin_approved":{"type":"boolean"},"marvin_processed":{"type":"boolean"},"organization_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"original_comment":{"type":"text","analyzer":"instantsearch","search_analyzer":"whitespace_lowercase"},"score":{"type":"integer"},"segments":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"supervisor_approved":{"type":"boolean"},"tenants":{"properties":{"tenant_220":{"properties":{"tag_ids":{"type":"keyword"},"tags":{"type":"keyword"}}}}},"term_vector_comment":{"type":"text","store":true,"term_vector":"with_positions_offsets_payloads","analyzer":"fulltext_analyzer"},"themes":{"type":"nested","properties":{"name":{"type":"keyword"},"parent":{"type":"keyword"},"sentiment":{"type":"integer"}}},"themes_verified":{"type":"boolean"},"type":{"type":"keyword"},"updated_at":{"type":"date"},"user_meta":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}
We have 1m +- docs into our database, and we want to have higt performance to index this docs. We build cluster with 4 data nodes (4g 4g), 2 masters (2g 2g), and 2 clients (1g 1g), all this we are running into kubernetes.
Then we create an erlang client, which gives us full power for parallel requests.
All looks good, but after 890k, we start to see, that new docs / requests does't create new documents. But response from bulk requests without any errors.
But when some another documents come to another indexes (we have > 400 indexes) we have this error:
[429] {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[es-data-300-3][10.244.12.220:9300][indices:data/write/bulk[s]]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of processing of [31516653][indices:data/write/bulk[s][p]]: request: BulkShardRequest
So my question, how we can fix this error, and why we don't have 419 for bulk request? It's looks like a bug, because if we have 419, we can retry like logstash or apm products. Looks like a bug for 6.6.2.
Thanks!