# Is elastic Ingest pipelines resource intensive?

**URL:** <https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [July 11, 2023, 2:38am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049 "2023-07-11T02:38:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\
**Post date:** [July 11, 2023, 2:38am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/1 "2023-07-11T02:38:48Z")

</div>

Hi,  
I am currently using injest pipelines to enrich my document before it wrote into index.  
I am wondering if this process would have a potenial hugh resouce(heap ram or cpu) comsumed for my Elasticsearch node behind the scene.

Below is the processors that composes my injest pipeline

```auto
PUT _ingest/pipeline/traces-apm@custom
{
  "processors": [
    {
      "script": {
        "source": """
       
        if(ctx.labels.region == null){
          ctx.db_tag=null
        } else if(ctx.labels.deviceId == null){
           ctx.db_tag=null
        } else{
           ctx.db_tag = ctx.labels.database_name + '-' + ctx.labels.region +'-' + ctx.labels.deviceId
        }
       
       """
      }
    },
    {
      "enrich": {
        "description": "Add extra data based on 'unique_db_tag'",
        "policy_name": "acav_limit_v1_policy",
        "field": "db_tag",
        "target_field": "unique_tag_v2",
        "max_matches": "1"
      }
    },
    {
      "set": {
        "field": "unique_tag_v2.pa_key",
        "value": "{{{labels.pa_key}}}"
      }
    },
    {
      "set": {
        "field": "unique_tag_v2.project_id",
        "value": "{{{labels.project_id}}}"
      }
    },
    {
      "set": {
        "field": "unique_tag_v2.brand_id",
        "value": "{{{labels.brandId}}}"
      }
    },
    {
      "set": {
        "field": "unique_tag_v2.function_id",
        "value": "{{{labels.function_id}}}"
      }
    },
    {
      "set": {
        "field": "unique_tag_v2.codeNum",
        "value": "{{{labels.codeNum}}}"
      }
    }
  ]
}

```

The reason I have this question is because, Elasticsearch has to loop through every document to see if it match to my enrich policy, executing the script processor, and other set processor, and it sounds to have a lot of work for Elasticsearch since I am using this injest pipeline for my data stream, therefore, I am wondering if this could have a huge resource comsumed, if it does, is there any way I can optimize it?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 11, 2023, 3:33am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/2 "2023-07-11T03:33:38Z")

</div>

What version Elasticsearch are you on?

ingest pipeline have consume various amounts of CPU and RAM depending on the complexity on the ingest pipeline and what it does.

Things like poor regex can be very expensive.

A quick look at your pipeline it is fairly low complexity.

> [@JasonREC](#):
>
> Elasticsearch has to loop through every document to see if it match to my enrich policy

This is not the correct way to look at that ... Elastic does not "loop" through the enrich index to enrich the source it does a `term` lookup which is extremely fast and efficient in Elasticsearch...

So I would say your pipeline is fairly low resource intensive... run it .. test it.

You can run

`GET /_nodes/hot_threads`

to see what is taking up resources

---

<div class="post-metadata">

**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\
**Post date:** [July 11, 2023, 4:37am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/3 "2023-07-11T04:37:11Z")

</div>

Hi @stephenb ,

I am currently on 8.8 version. Good to know that my injest pipeline is in low complexity and thanks for pointing out that poor regex will make intensive resouce comsumed, I will pay attension in future query writting.

And I run GET /\_nodes/hot\_threads, it looks very normal 😃  
1.4% [cpu=1.4%, idle=98.6%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[instance-0000000004][transport\_worker][T#1]'

I am a bit intersted about the term look up regarding how Elastic enrich the document. I am not sure if I understand the flow correctly  
I found this picture in offical document.

 ![flowchart](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fcb57944084d4cdfc1b8b67eab235b71b7833ab7.png)

For example, my incoming document has a field called email and my source index also has an email field, as long as the incoming document email value match the email value in the source index, this document will be write into an index called "enrich index", I assum that would look like something like below inside the "enrich" index.  
 ![termlookup](https://us1.discourse-cdn.com/elastic/original/3X/8/2/8283e37620e100ee185e9c46a9ac0245d9f37bd6.png)

Then Elastic perform a **term** look up which like jump exactly to the document location, for example, the docuemnt just be indexed into enrich index, and then elastic take it out for the rest of processor execution. Because of this, it is extremely fast.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 11, 2023, 5:42am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/4 "2023-07-11T05:42:44Z")

</div>

The first picture is correct

> [@JasonREC](#):
>
> Then Elastic perform a **term** look up which like jump exactly to the document location, for example, the docuemnt just be indexed into enrich index, and then elastic take it out for the rest of processor execution. Because of this, it is extremely fast.

The 2nd picture is doc values which are not really used for this part. 🙂

The enrich index is compacted and optimized for the term lookup.  
Pretty much depends on how deep you want to get but a term lookup uses the inverted index to find the record extremely quickly. Doc Values are used for aggregations sorting not really lookup.

So yeah..its fast and efficient... especially for reasonable-sized enrich indices (lets say 100Ks to Low Millions)

There are lots of data structures in elastic it may take a while to figure it all out.

---

<div class="post-metadata">

**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\
**Post date:** [July 11, 2023, 6:36am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/5 "2023-07-11T06:36:10Z")

</div>

@stephenb , Thanks for the quick reply!

I see, if it is using inverted index similar to the picture below, it makes so much sense why it can be so quick  
 ![invertindex](https://us1.discourse-cdn.com/elastic/original/3X/6/6/66d664c9ed0db0deb8fb9fc8d8f2d88d8d698061.png)

pretty much, in the chart, term could the email field value, and doc Id can be those matched email document's ID

Appreciated for answering my question, it is very helpful!

---

<div class="post-metadata">

**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\
**Post date:** [July 12, 2023, 8:40am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/6 "2023-07-12T08:40:45Z")

</div>

@stephenb

Sorry, I have one more question that I am curious, let say we have 500000 different email address in the **source index** , and every incoming document contains an email address, does Elastic search has to loop through the soruce index and compare one by one with the incoming document to find the matched email so that then it can injest the data to the incoming data?

if this is the case, would it be resource intensive when my source index's size keep going up?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 12, 2023, 1:16pm UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/7 "2023-07-12T13:16:20Z")

</div>

I am not following.. elastic does not loop through anything.

If by source index you mean the source of the enrich index i.e  
The look up index no it uses the term query / inverted index as discussed to find the matching enrich data very efficiently

As the picture shows as each new document comes in the enrich / lookup happens... Very fast.

---

<div class="post-metadata">

**Author:** ![JasonREC](https://avatars.discourse-cdn.com/v4/letter/j/858c86/32.png) [@JasonREC](https://discuss.elastic.co/u/JasonREC)\
**Post date:** [July 13, 2023, 4:30am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/8 "2023-07-13T04:30:05Z")

</div>

@stephenb

Thank you! I think I got it now

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 10, 2023, 4:30am UTC](https://discuss.elastic.co/t/is-elastic-ingest-pipelines-resource-intensive/338049/9 "2023-08-10T04:30:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.