How do I detect that a previous processor has succeeded in an ingest pipeline?

[jerrac (David Reagan)]

I'm using Filebeat, plus hints based auto discovery, to get my Docker Swarmt container Apache logs passed through the built in Apache logs ingest pipelines.

The thing is, that several of my apache containers are also outputting logs from Symfony and/or Drupal.

Those logs, obviously, don't match any Apache grok pattern and cause grok parse failure messages.

To fix that, I have a final ingest pipeline configured. In it I detect if there is a grok parse error, if the doc is tagged for symfony or drupal, and then run it through my own grok pattern.

That works just fine, but I'm still left with the error.message field claiming there is still a grok error.

I'd like to go ahead and just drop that field when my custom grok processor succeeds.

Where I'm stuck is that I'm not seeing an obvious way to detect if my grok processors have succeeded so I can run a remove processor on the error.message field.

I could just remove the field all the time, but that would prevent me from finding the errors I need to fix.

I also tried adding a tag in the grok processor, but for some reason those tags don't actually get added to the tags field. (A long time ago I ran across this and someone explained it, but I can't recall what they said and didn't find the topic in a quick search...)

Anyone have any ideas?

Thanks!

[leandrojmp (Leandro Pereira)]

@jerrac

Where I'm stuck is that I'm not seeing an obvious way to detect if my grok processors have succeeded

You can only do the inverse, tag it if it fails.

You would need to use the on_failure option and add a specific tag for each grok you have, this way you can know which one is failing.

Something like this:

        "on_failure": [
          {
            "append": {
              "field": "tags",
              "value": "grok-abcd-123-failed"
            }
          }
        ]

[jerrac (David Reagan)]

Tagging on failure is what I ended up doing. Though I ended up creating a custom pipeline for each of my log types. That lets me have a bit more control. Thanks @leandrojmp for the tip.