# Enrich Processor missing documents **URL:** \ **Category:** Elasticsearch\ **Tags:** ingest-pipeline\ **Created:** [March 29, 2023, 2:14pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843 "2023-03-29T14:14:56Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![FKarraz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fkarraz/32/80628_2.png) [@FKarraz](https://discuss.elastic.co/u/FKarraz)\ **Post date:** [March 29, 2023, 2:14pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843/1 "2023-03-29T14:14:56Z")
Hi, i have several ingest pipelines that has quite large processor configured in it. Each pipeline for each Data Stream. For example, pipeline "2g\_names" works with "raw\_kpi\_2g\__" (raw\_kpi\_2g\_1) Data Stream, "3g\_names" for "raw\_kpi\_3g\__" (raw\_kpi\_3g\_1, raw\_kpi\_3g\_2, raw\_kpi\_3g\_3 & raw\_kpi\_3g\_4) and "4g\_names" works with "raw\_kpi\_4g\_\*" (raw\_kpi\_4g\_1, raw\_kpi\_4g\_2, raw\_kpi\_4g\_3, raw\_kpi\_4g\_4 & raw\_kpi\_4g\_5). The Elasticsearch version is 7.12.1 This is one of the pipelines ("3g\_names"): ```auto { "3g_names" : { "description" : "Enrich 3G data with names", "processors" : [ { "enrich" : { "field" : "RNC.dn", "policy_name" : "mo_name", "target_field" : "RNC", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "WBTS.dn", "policy_name" : "mo_name", "target_field" : "WBTS", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "WCEL.dn", "policy_name" : "mo_name", "target_field" : "WCEL", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "WBTS.name", "policy_name" : "location_data", "target_field" : "location", "ignore_missing" : true, "ignore_failure" : true } }, { "set" : { "field" : "location.name", "value" : "{{location.shortName}}", "ignore_failure" : true } }, { "remove" : { "field" : "location.shortName", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "location.name", "policy_name" : "tech_correlation", "target_field" : "general", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "location.name", "policy_name" : "jefaturas_data_v2", "target_field" : "jefaturas", "ignore_missing" : true, "ignore_failure" : true } }, { "enrich" : { "field" : "WBTS.name", "policy_name" : "site_name", "target_field" : "temp", "ignore_missing" : true, "ignore_failure" : true } }, { "set" : { "field" : "location.siteName", "value" : "{{temp.siteName}}", "ignore_empty_value" : true, "ignore_failure" : true } }, { "remove" : { "field" : "temp", "ignore_missing" : true, "ignore_failure" : true } } ] } } ``` The pipeline actually works when i simulate, but when i send a large bulk request to index data, it fails with following error: ```auto { "processor_results":[ { "processor_type":"enrich", "status":"error_ignored", "ignored_error":{ "error":{ "root_cause":[ { "type":"es_rejected_execution_exception", "reason":"Could not perform enrichment, enrich coordination queue at capacity [1024/1024]" } ], "type":"es_rejected_execution_exception", "reason":"Could not perform enrichment, enrich coordination queue at capacity [1024/1024]" } } } ] } ``` Here are some statistics of the node (we have only 1) of the last 7 days: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/6/16f331bb079a3f798fe625203a2a27801572c071.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/a/ba49d8668f8b19012cacaaaf40d71cd51df350c4.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/5/c541e08b7a15627225530070fbf042b054d08dae.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d5499f4a792f8dc9f552d17c909cc919003ea69.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/06bca4d62927d08f6d7efd102c467dce8615d6ee.png) Here are some statistics one of the behind index (writing index) of one DataStream: ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/2/0244de54fc42c962d67d1bfd2dc40fea804616ee.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/4/94ab1a6a69fd45cffd01f7defc47c3c2be241a29.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d3a5c42058f0b5a3272a2909118e1dee0c2efee.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/8/387f86e0bea98ab0e5945da4ad4dccbf9ba7b5cf.png) I couldn't get the indexing rate of each data stream, if someone can tell me how to make the query, I will be happy to provide that information. What approach could be taken to this situation in order to solve this problem? Regards. ---
**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\ **Post date:** [March 30, 2023, 7:48pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843/2 "2023-03-30T19:48:54Z")
You might benefit from the enrich cache added in 7.16.0 -- [Add enrich node cache by martijnvg · Pull Request #76800 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/76800). Are you able to upgrade? I see some of the settings documented under `Enrich settings` at [Edit Elasticsearch user settings | Elasticsearch Service Documentation | Elastic](https://www.elastic.co/guide/en/cloud/current/ec-add-user-settings.html) (I'm not sure why they are there and not in the Elasticsearch documentation, but they are Elasticsearch settings). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 30, 2023, 10:20pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843/3 "2023-03-30T22:20:10Z")
> [@FKarraz](#): > > The Elasticsearch version is 7.12.1 > [@Keith\_Massey](#): > > Are you able to upgrade? I want to second Keith's comments. That version is [EOL](https://www.elastic.co/support/eol) and no longer supported, you should be looking to upgrade as a matter of urgency. ---
**Author:** ![FKarraz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fkarraz/32/80628_2.png) [@FKarraz](https://discuss.elastic.co/u/FKarraz)\ **Post date:** [April 3, 2023, 12:24pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843/4 "2023-04-03T12:24:30Z")
Thanks @Keith_Massey & @warkolm for the reply. Do you have some guide or recommended approach to upgrade the elasticsearch (that is running on a ducker container) from 7.12.1 to 7.17.9? It's necessary to take aware for something or with just changing the tag version of docker image will work? Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 1, 2023, 12:25pm UTC](https://discuss.elastic.co/t/enrich-processor-missing-documents/328843/5 "2023-05-01T12:25:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.