# How do I detect that a previous processor has succeeded in an ingest pipeline?

**URL:** <https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240>\
**Category:** Elasticsearch\
**Tags:** docker, ingest-pipeline\
**Created:** [October 17, 2023, 6:39pm UTC](https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240 "2023-10-17T18:39:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [October 17, 2023, 6:39pm UTC](https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240/1 "2023-10-17T18:39:16Z")

</div>

I'm using Filebeat, plus hints based auto discovery, to get my Docker Swarmt container Apache logs passed through the built in Apache logs ingest pipelines.

The thing is, that several of my apache containers are also outputting logs from Symfony and/or Drupal.

Those logs, obviously, don't match any Apache grok pattern and cause grok parse failure messages.

To fix that, I have a final ingest pipeline configured. In it I detect if there is a grok parse error, if the doc is tagged for symfony or drupal, and then run it through my own grok pattern.

That works just fine, but I'm still left with the `error.message` field claiming there is still a grok error.

I'd like to go ahead and just drop that field when my custom grok processor succeeds.

Where I'm stuck is that I'm not seeing an obvious way to detect if my grok processors have succeeded so I can run a remove processor on the `error.message` field.

I could just remove the field all the time, but that would prevent me from finding the errors I need to fix.

I also tried adding a tag in the grok processor, but for some reason those tags don't actually get added to the tags field. (A long time ago I ran across this and someone explained it, but I can't recall what they said and didn't find the topic in a quick search...)

Anyone have any ideas?

Thanks!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [October 17, 2023, 6:47pm UTC](https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240/2 "2023-10-17T18:47:23Z")

</div>

> [@jerrac](#):
>
> Where I'm stuck is that I'm not seeing an obvious way to detect if my grok processors have succeeded

You can only do the inverse, tag it if it fails.

You would need to use the `on_failure` option and add a specific tag for each grok you have, this way you can know which one is failing.

Something like this:

```auto
        "on_failure": [
          {
            "append": {
              "field": "tags",
              "value": "grok-abcd-123-failed"
            }
          }
        ]

```

---

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [October 18, 2023, 4:20pm UTC](https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240/3 "2023-10-18T16:20:47Z")

</div>

Tagging on failure is what I ended up doing. Though I ended up creating a custom pipeline for each of my log types. That lets me have a bit more control. Thanks @leandrojmp for the tip. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 15, 2023, 4:21pm UTC](https://discuss.elastic.co/t/how-do-i-detect-that-a-previous-processor-has-succeeded-in-an-ingest-pipeline/345240/4 "2023-11-15T16:21:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.