# Usually data are not enriched **URL:** \ **Category:** Elasticsearch\ **Created:** [October 12, 2022, 12:21pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435 "2022-10-12T12:21:27Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [October 12, 2022, 12:21pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/1 "2022-10-12T12:21:27Z")
Hi, Does exists some situactions, when data are not enriched ? (i'm excluding scenario when there are no matching data) Example : Can be skipped for perfomance reasons? It is possible that enrich processor in pipeline can be skipped? By adding tags can I exclude scenatio with skipping enrich processor ? Flow : Logs -\> Logstash -\> Elasticsearch pipeline (where enrich processor exists) -\> Elasticsearch Index Indexing speed(enriched index) : approx : 80k in 45minutes Elasticsearch 7.17 ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 12, 2022, 1:58pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/2 "2022-10-12T13:58:40Z")
You can configure your processor to use a conditional and it will only run if the conditional is true. Check this [part of the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/master/ingest.html#conditionally-apply-pipelines). Also, since you are using Logstash, and depending on what you are enrich, it would be much easier and fast to enrich the data in Logstash. ---
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [October 13, 2022, 7:21am UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/3 "2022-10-13T07:21:25Z")
Thank you for reply. OK, i understood that using if statements can skip enrich processor in ingest pipeline What I want to enrich? Example : Index : "products" contains : Name of product, type of product , ID, and color Index : "pricing" contains : Price, tax, ID I want to enrich "products" with Price, using enrich processor in elasticsearch pipeline Additionally i understood that i can do it in logstash When enrich processor in elastic pipeline can be skipped ? (excluding : if statements, there are no matching data, for enrichment i want to use elastic pipeline with enrich processor) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 13, 2022, 12:52pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/4 "2022-10-13T12:52:03Z")
The ingest pipeline will run the processors in the order they are configured, if you have an `enrich` processor in the ingest pipeline for your `products` index, it will be executed for every event. If you want to skip a processor you need a conditional based on some kind of data, there is no other way to skip the execution of a processor. Also, the enrich processor is recommended for static data, if you will need to constantly update your source index, you will need to manually call the excute api to update the data of the enrich index every time the source index is updated. ---
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [October 17, 2022, 5:32am UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/5 "2022-10-17T05:32:20Z")
OK, so.. What will happen when i'm not update data of the enrich index after adding some data(in source index)? Enrich processor will run but without results? By update data of the enrich index, you mean call /\_enrich/policy/name\_of\_policy/\_execute ? so... if I exclude if statements, and enrich processor will be defined in pipeline, then it will ALWAYS execute? it cannot be skipped? or.. When data can be not enriched? (excluding not updating enrich index after adding some data) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 17, 2022, 3:04pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/6 "2022-10-17T15:04:06Z")
> [@Marcin\_Frankiewicz](#): > > so.. What will happen when i'm not update data of the enrich index after adding some data(in source index)? > Enrich processor will run but without results? > By update data of the enrich index, you mean call /\_enrich/policy/name\_of\_policy/\_execute ? If you add new data to the source index of your enrich policy, this new data will only be available after you run a new `_execute` on the policy, this will create a new enrich index. > [@Marcin\_Frankiewicz](#): > > so... if I exclude if statements, and enrich processor will be defined in pipeline, then it will ALWAYS execute? it cannot be skipped? If you do not have an if conditional in your enrich processer it will **always** be executed. As I said in the previous answer the processors are executed in the order they are configured in the ingest pipeline and they are always executed, if you want to skip some processors you need to use a conditional on that processor to check if it can be executed or not, in this case the conditional will be always executed. > [@Marcin\_Frankiewicz](#): > > When data can be not enriched? (excluding not updating enrich index after adding some data) If there is no match in the enrich processor, the data will not be enriched, but the processor will always be executed. ---
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [November 2, 2022, 2:55pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/7 "2022-11-02T14:55:57Z")
How many replicas should have .enrich-INDEXNAME indicle ? Should be a sum of : Hot + Warm + Ingest ? or maybe : only ingest nodes? What roles should have HOT/WARM/Ingest nodes? (3x HOT(drt), 3x WARM(drt), 6x Ingest(di) ) I'm trying to deal with situation that usually data are not enriched, but not because there is no matching data... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 2, 2022, 3:02pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/8 "2022-11-02T15:02:53Z")
> [@Marcin\_Frankiewicz](#): > > How many replicas should have .enrich-INDEXNAME indicle ? You should leave it using the default configuration, If i'm not wrong it will auto-expand to every data node, or at elast every node with a `data_content` role. > [@Marcin\_Frankiewicz](#): > > I'm trying to deal with situation that usually data are not enriched, but not because there is no matching data... What situation? Provide more context. ---
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [November 8, 2022, 12:13pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/9 "2022-11-08T12:13:18Z")
There are two independent sources gathered by Logstash (two indexes: index1, index2) Every source, has **own** Logstash pipeline. Index1 looks simple. Logstash crawls CSV files, there are some filter rules like, csv, translate,dissect and fingerprint fingerprint is calculated from two fields concatenate\_sources =\> true Logstash in output, has interesting options doc\_as\_upsert =\> true document\_id =\> "%{fingerprint}" Also here is defined pipeline where document will be sent Ingest pipeline has few processor like date, grok, date \_enrich/policy is executed 3x per day ```auto "match_field" : "index1.number", "enrich_fields" : [ "geo.region_iso_code_old", "index1.commune.id", "index1.company.id", "index1.company.name", "index1.service" ] ``` CSV file is created once per day (~00:00AM) ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/6/d66dc541d7a74f98a8e61743bd7544ab301a2a9d.png) Above image shows that ~98% records are **updated** Index2 are more complex. Logstash crawls CSV files, there are some filter rules like, csv, dissect, mutate, and it is sent to ingest pipeline. That pipeline has many sub-pipelines, but in one of them contains enrich processor, which depends on index1 ```auto "enrich" : { "tag" : "index1 b", "ignore_missing" : true, "policy_name" : "index1", "field" : "tmp.enrich_pl.value", "target_field" : "tmp.enrich_pl.b", "max_matches" : "1" } ``` Final effect ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/4/94b1ededd7c141e2fb19d752ef3663737b426857.png) ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [November 8, 2022, 12:52pm UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/10 "2022-11-08T12:52:53Z")
I'm sorry, but i didn't get what is the issue with just that information. You didn't share what your data looks like, you didn't share what the data of your enrich index looks like, for example, in your last screenshot you shared a field named `b.company.name`, where this comes from? It is not possible to know what may be the issue from what you shared. You need to share some sample data of both your index and enrich data so it is possible to someone to try to replicate your issue. Can you share an example of a document that should've be enriched, but wasn't, and also the data from the enrich index? ---
**Author:** ![Marcin\_Frankiewicz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcin_frankiewicz/32/97650_2.png) [@Marcin\_Frankiewicz](https://discuss.elastic.co/u/Marcin_Frankiewicz)\ **Post date:** [December 2, 2022, 10:04am UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/11 "2022-12-02T10:04:59Z")
I have found resolution for my problem. That was an error in ingest pipeline. In Kibana i saw phone number after their cleanup(done by other pipeline)(example : 00672794579), but while enrich processor was called, this phone number contained some unnecessary data(example : C43500672794579). So, phone number could not be matched with enrich index I have added gsub processor **before** enrich processor, and now it is working fine Thanks for your help! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 30, 2022, 10:05am UTC](https://discuss.elastic.co/t/usually-data-are-not-enriched/316435/12 "2022-12-30T10:05:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.