Drop events if regex does not match in ElasticSearch

Elastic Stack Elasticsearch
1

I'm getting events from Filebeat directly to ElasticSearch for metrics live viewing in the Observability menu, however I can't drop out the events I'm not interested like I would be able to using Logstash. I'm receiving the info from Filebeat to ElasticSearch using a pipeline that first filters the events using a custom field and redirecting to the appropriate pipeline accordingly. In the second pipeline, I need to apply regular expressions to drop anything that is not what I want. What I have tried:

PUT _ingest/pipeline/eventtwoline_pipeline
{
  "description": "Ingest for two line live events.",
  "processors": [
    {
      "drop": {
        "if" : "ctx.message != /^(dog|cat)/"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'dog_pipeline'",
        "if": "ctx.message =~ /^dog/",
        "name": "dog_pipeline"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'cat_pipeline'",
        "if": "ctx.message =~ /^cat/",
        "name": "cat_pipeline"
      }
    }
  ]
}

And with painless:

PUT _ingest/pipeline/eventtwoline_pipeline
{
  "description": "Ingest for two line live events.",
  "processors": [
    {
      "drop": {
        "if": """
           String m = ctx['message'];
              if (!m.matches("^(dog|cat)")){
                 return true;
              }
        
           return false;
        """
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'dog_pipeline'",
        "if": "ctx.message =~ /^dog/",
        "name": "dog_pipeline"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'cat_pipeline'",
        "if": "ctx.message =~ /^cat/",
        "name": "cat_pipeline"
      }
    }
  ]
}

But can't seem to make it work. On the first one, the document is always null when I use the _simulate API, possibly because there is no operator for this instead of =~, and the second one, I've tried a lot of different things and always results in compilation errors. Any suggestions?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.