# Drop events if regex does not match in ElasticSearch

**URL:** <https://discuss.elastic.co/t/drop-events-if-regex-does-not-match-in-elasticsearch/274803>\
**Category:** Elasticsearch\
**Created:** [June 2, 2021, 10:31pm UTC](https://discuss.elastic.co/t/drop-events-if-regex-does-not-match-in-elasticsearch/274803 "2021-06-02T22:31:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pbreda](https://avatars.discourse-cdn.com/v4/letter/p/c4cdca/32.png) [@pbreda](https://discuss.elastic.co/u/pbreda)\
**Post date:** [June 2, 2021, 10:31pm UTC](https://discuss.elastic.co/t/drop-events-if-regex-does-not-match-in-elasticsearch/274803/1 "2021-06-02T22:31:49Z")

</div>

I'm getting events from Filebeat directly to ElasticSearch for metrics live viewing in the Observability menu, however I can't drop out the events I'm not interested like I would be able to using Logstash. I'm receiving the info from Filebeat to ElasticSearch using a pipeline that first filters the events using a custom field and redirecting to the appropriate pipeline accordingly. In the second pipeline, I need to apply regular expressions to drop anything that is not what I want. What I have tried:

```auto
PUT _ingest/pipeline/eventtwoline_pipeline
{
  "description": "Ingest for two line live events.",
  "processors": [
    {
      "drop": {
        "if" : "ctx.message != /^(dog|cat)/"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'dog_pipeline'",
        "if": "ctx.message =~ /^dog/",
        "name": "dog_pipeline"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'cat_pipeline'",
        "if": "ctx.message =~ /^cat/",
        "name": "cat_pipeline"
      }
    }
  ]
}

```

And with painless:

```auto
PUT _ingest/pipeline/eventtwoline_pipeline
{
  "description": "Ingest for two line live events.",
  "processors": [
    {
      "drop": {
        "if": """
           String m = ctx['message'];
              if (!m.matches("^(dog|cat)")){
                 return true;
              }
        
           return false;
        """
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'dog_pipeline'",
        "if": "ctx.message =~ /^dog/",
        "name": "dog_pipeline"
      }
    },
    {
      "pipeline": {
        "description": "If 'ctx.message' matches, use 'cat_pipeline'",
        "if": "ctx.message =~ /^cat/",
        "name": "cat_pipeline"
      }
    }
  ]
}

```

But can't seem to make it work. On the first one, the document is always null when I use the `_simulate` API, possibly because there is no operator for this instead of =~, and the second one, I've tried a lot of different things and always results in compilation errors. Any suggestions?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2021, 10:32pm UTC](https://discuss.elastic.co/t/drop-events-if-regex-does-not-match-in-elasticsearch/274803/2 "2021-06-30T22:32:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.