# Exist Elasticsearch ingest pipeline if specific field exists **URL:** \ **Category:** Elasticsearch\ **Tags:** ingest-pipeline\ **Created:** [March 1, 2022, 9:01pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563 "2022-03-01T21:01:12Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [March 1, 2022, 9:01pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/1 "2022-03-01T21:01:12Z")
In an Elasticsearch Ingest Pipeline, how can I validate whether a specific field exists, and exit the pipeline immediately without processing if it does? ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [March 2, 2022, 5:21am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/2 "2022-03-02T05:21:39Z")
I suppose [Fail processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/fail-processor.html) could. ---
**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Post date:** [March 2, 2022, 6:11am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/3 "2022-03-02T06:11:24Z")
Each process has an if parameter to Conditionally execute the processor. For example，only doc has field k1，then add filed res from k1'value. ```auto POST _ingest/pipeline/_simulate { "pipeline": { "processors": [ { "set": { "if": "ctx.containsKey(\"k1\")", "field": "res", "copy_from": "k1" } } ] }, "docs": [ { "_source": { "k1":"1" } }, { "_source": { "k2":"1" } } ] } ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 2, 2022, 6:26am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/4 "2022-03-02T06:26:18Z")
You just write a top level pipeline and a pipeline that does the work. Look at [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-apply-pipelines) In your case the condition to call the work pipeline would look something like `"if": "ctx?.myfield != null",` If the field does not exist it will not execute. ---
**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Post date:** [March 2, 2022, 6:32am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/6 "2022-03-02T06:32:29Z")
Can you tell me what **"?"** after ctx means below？thanks ```auto "if": "ctx?.myfield != null" ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [March 2, 2022, 6:35am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/7 "2022-03-02T06:35:52Z")
It's a "null Safety" check part of the syntax Read this [section](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-run-processor) > Incoming documents often contain object fields. If a processor script attempts to access a field whose parent object does not exist, Elasticsearch returns a NullPointerException. To avoid these exceptions, use null safe operators, such as ?., and write your scripts to be null safe. > > For example, ctx.network?.name.equalsIgnoreCase('Guest') is not null safe. ctx.network?.name can return null. Rewrite the script as ---
**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\ **Post date:** [March 2, 2022, 6:37am UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/8 "2022-03-02T06:37:38Z")
thanks a lot ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [March 2, 2022, 2:00pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/9 "2022-03-02T14:00:37Z")
> [@Tomo\_M](#): > > I suppose [Fail processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/fail-processor.html) could. I like this, but wish it could exit silently, without throwing an exception. It seems to me as if there should be an analogous `exit` processor. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [March 2, 2022, 2:01pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/10 "2022-03-02T14:01:40Z")
> [@casterQ](#): > > Each process has an if parameter to Conditionally execute the processor. > For example，only doc has field k1，then add filed res from k1'value. This is what I'm currently doing. However, it's a fairly long processor, which only executes if a specific field is present. ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [March 2, 2022, 2:04pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/11 "2022-03-02T14:04:29Z")
How about [Drop processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/drop-processor.html)? It is possible to use `if` conditional in the processor. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [March 2, 2022, 2:05pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/12 "2022-03-02T14:05:41Z")
> [@stephenb](#): > > You just write a top level pipeline and a pipeline that does the work. > > Look at [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-apply-pipelines) > > In your case the condition to call the work pipeline would look something like > > `"if": "ctx?.myfield != null",` > > If the field does not exist it will not execute. I hadn't thought of this - it may be what I do. Just out of curiosity, is there any significant overhead from calling a second ingest pipeline? I see that a few of the filebeat module pipelines do this (i.e., the `elasticsearch` pipelines). ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [March 2, 2022, 2:07pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/13 "2022-03-02T14:07:34Z")
> [@Tomo\_M](#): > > How about [Drop processor](https://www.elastic.co/guide/en/elasticsearch/reference/master/drop-processor.html)? > It is possible to use `if` conditional in the processor. I was unclear in my original post. I need to keep the document, no matter what, but only need to continue the pipeline if the specific field is not present. This would drop the document. ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [March 2, 2022, 2:30pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/14 "2022-03-02T14:30:22Z")
Now I understand what you want and found a trick. Raise failure by internal fail processor and catch it by `on_failure` at the top of the pipeline definition. Specifying some null processor (set processor here) `on_failure`, the pipeline exits immediately at meeting the if conditional of the fail processor and index the document at the moment. Please see ["Handling pipeline failures"](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#handling-pipeline-failures) for the behavior. ```auto POST /_ingest/pipeline/_simulate { "docs":[ { "_source":{ "foo":"baa" } },{ "_source":{ "foo":"foo" } } ], "pipeline": { "processors": [ {"fail":{ "if":"ctx.foo=='baa'", "ignore_failure": false, "message":"***" }}, { "set":{ "field": "following", "value": "processors" } } ], "on_failure":[{ "set":{ "if":"false", "field": "null", "value": "null" } }] } } ``` Then you get: ```auto { "docs" : [ { "doc" : { "_index" : "_index", "_type" : "_doc", "_id" : "_id", "_source" : { "foo" : "baa" }, "_ingest" : { "timestamp" : "2022-03-02T14:30:13.239791493Z" } } }, { "doc" : { "_index" : "_index", "_type" : "_doc", "_id" : "_id", "_source" : { "foo" : "foo", "following" : "processors" }, "_ingest" : { "timestamp" : "2022-03-02T14:30:13.239795055Z" } } } ] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 30, 2022, 2:30pm UTC](https://discuss.elastic.co/t/exist-elasticsearch-ingest-pipeline-if-specific-field-exists/298563/15 "2022-03-30T14:30:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.