# Logstash elasticsearch output plugin filtering

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795>\
**Category:** Logstash\
**Created:** [April 4, 2018, 5:27pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795 "2018-04-04T17:27:15Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [April 4, 2018, 5:27pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/1 "2018-04-04T17:27:15Z")

</div>

The application are sending logs in JSON format with some additional tags for example logType, [attribute][APPLICATION] etc. and we want apply filter on logstash based on those tags so that logstash will not send some type of logs to ES cluster. The conditions are

- if the logs from all application and the type is access then drop it
- if the application is app1 and log type is redirect then drop
- and the log type is missing or is error then sent it to dynamically created ES indices.

Here is the configuration file

```
output {

    if [logType] != "access" or ( [attribute][APPLICATION] == "app1" and [logType] != "redirect" ) {

      if [logType] =~ "error" or ![logType] {
        amazon_es {
          hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
          region => "us-west-1"
          index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-error-%{+YYYY.MM.dd}"
        }
      }

      else {
        amazon_es {
          hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
          region => "us-west-1"
          index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[logType]}-%{+YYYY.MM.dd}"
        }
     }

  }

}

```

But logstash is not sending any logs. Please help.

Ferdous Shibly

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 4, 2018, 5:45pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/2 "2018-04-04T17:45:43Z")

</div>

> [@Ferdous\_Shibly](#):
>
> ```auto
> [logType] =~ "error"
> 
> ```

The right-hand side of pattern-match expressions needs to be a pattern, not a literal string; you'll want the following instead.

```auto
[logType] =~ /error/

```

Would you care to share your logstash version? I'd like to follow up and file a bug to make this easier to detect.

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [April 4, 2018, 5:51pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/3 "2018-04-04T17:51:08Z")

</div>

@yaauie Thanks for the info. So the current configuration looks like this

```
output {

if [logType] != /access/ or ( [attribute][APPLICATION] =~ /app1/ and [logType] != /redirect/ ) {

  if [logType] =~ /error/ or ![logType] {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-error-%{+YYYY.MM.dd}"
    }
  }

  else {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[logType]}-%{+YYYY.MM.dd}"
    }
 }

  }

}

```

But still I don't see any logs in ES. We are using logstash-5.6.7-1.noarch.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 4, 2018, 6:58pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/4 "2018-04-04T18:58:43Z")

</div>

> [@yaauie](#):
>
> The right-hand side of pattern-match expressions needs to be a pattern, not a literal string; you'll want the following instead.

A literal string works just fine in 6.2, even if it uses regexp patterns.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 4, 2018, 9:54pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/5 "2018-04-04T21:54:09Z")

</div>

> [@Ferdous\_Shibly](#):
>
> ```auto
> if [logType] != /access/ or ( [attribute][APPLICATION] =~ /selene/ and [logType] != /redirect/ ) {
> # ...
> }
> 
> ```

this `if` statement doesn't have an `else` clause, so documents that do not match are not getting output anywhere; what is the shape of the documents that are failing to match?

```auto
output {
  # ...
  else {
    stdout { codec => json }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [April 9, 2018, 5:22pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/6 "2018-04-09T17:22:11Z")

</div>

thanks @yaauie. is it possible to send the logs to null which documents that do not match? The problem which we are facing now are -

- some of out applications are sending lots of access logs and redirect logs so that ES cluster is getting full more frequently. So we want not to send access logs (and some redirect logs) to ES cluster which we don't need.
- but we want to send all logs to S3 buckets.

The logstash pipeline was working properly but we don't know how to filter those not to send Elasticsearch only.

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 9, 2018, 8:11pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/7 "2018-04-09T20:11:39Z")

</div>

failing to provide an `else` clause in the output is functionally equivalent to providing an `else` with a null-output:

```auto
 else {
  null {}
 }

```

The worry here, is it's sometimes tricky to get the logic "right", especially with negations and groupings in `if`-clauses, so at least while developing/prototyping, I tend to _always_ include an else-clause to stdout until I'm happy that everything that is going to stdout is stuff I don't want in my other outputs.

Typically, when writing a complex pipeline, I add tags to events that explicitly match the patterns I'm looking for, and then use the presence of tags to determine which output(s) to send to; in your case, it looks like you have two separate outputs to handle the coercion of a missing `[logType]`, so I would do something like the following:

```auto
filter {
  # ..

  # ensure we have a derived logType in our @metadata, even if the
  # original log message didn't include one.
  if ![logType] {
    mutate { "add_field" => {"[@metadata][logType]" => "error"} } 
  } else {
    mutate { "add_field" => {"[@metadata][logType]" => "%{[logType]}"} }
  }

  # ...

  # explicitly flag types of logs we do want in out output with the `export` tag:
  if [logType] != "access" {
    mutate { add_tag => "export" }
  }
  if [attribute][application] =~ /app1/ and [logType] != "redirect" {
    mutate { add_tag => "export" }
  }
}
output {
  if "export" in [tags] {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[@metadata][logType]}-%{+YYYY.MM.dd}"
    }
  } else {
    # although the null-output isn't necessary, it may be helpful when combined
    # with the [X-Pack Pipeline Viewer][1], which shows consumption rates of
    # pipeline nodes in a web UI. 
    # [1]: https://www.elastic.co/guide/en/logstash/6.x/logstash-pipeline-viewer.html
    null {}
  }
}

```

---

<div class="post-metadata">

**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\
**Post date:** [April 9, 2018, 8:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/8 "2018-04-09T20:52:00Z")

</div>

@yaauie looks promising. I am going to change our configuration and inform you tomorrow.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2018, 8:52pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-output-plugin-filtering/126795/9 "2018-05-07T20:52:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.