Logstash elasticsearch output plugin filtering

Elastic Stack Logstash
1

The application are sending logs in JSON format with some additional tags for example logType, [attribute][APPLICATION] etc. and we want apply filter on logstash based on those tags so that logstash will not send some type of logs to ES cluster. The conditions are

  • if the logs from all application and the type is access then drop it
  • if the application is app1 and log type is redirect then drop
  • and the log type is missing or is error then sent it to dynamically created ES indices.

Here is the configuration file

output {

    if [logType] != "access" or ( [attribute][APPLICATION] == "app1" and [logType] != "redirect" ) {

      if [logType] =~ "error" or ![logType] {
        amazon_es {
          hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
          region => "us-west-1"
          index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-error-%{+YYYY.MM.dd}"
        }
      }

      else {
        amazon_es {
          hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
          region => "us-west-1"
          index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[logType]}-%{+YYYY.MM.dd}"
        }
     }

  }

}

But logstash is not sending any logs. Please help.

Ferdous Shibly

2

The right-hand side of pattern-match expressions needs to be a pattern, not a literal string; you'll want the following instead.

[logType] =~ /error/

Would you care to share your logstash version? I'd like to follow up and file a bug to make this easier to detect.

3

@yaauie Thanks for the info. So the current configuration looks like this

output {

if [logType] != /access/ or ( [attribute][APPLICATION] =~ /app1/ and [logType] != /redirect/ ) {

  if [logType] =~ /error/ or ![logType] {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-error-%{+YYYY.MM.dd}"
    }
  }

  else {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[logType]}-%{+YYYY.MM.dd}"
    }
 }

  }

}

But still I don't see any logs in ES. We are using logstash-5.6.7-1.noarch.

4

A literal string works just fine in 6.2, even if it uses regexp patterns.

1 Like
5

this if statement doesn't have an else clause, so documents that do not match are not getting output anywhere; what is the shape of the documents that are failing to match?

output {
  # ...
  else {
    stdout { codec => json }
  }
}
6

thanks @yaauie. is it possible to send the logs to null which documents that do not match? The problem which we are facing now are -

  • some of out applications are sending lots of access logs and redirect logs so that ES cluster is getting full more frequently. So we want not to send access logs (and some redirect logs) to ES cluster which we don't need.
  • but we want to send all logs to S3 buckets.

The logstash pipeline was working properly but we don't know how to filter those not to send Elasticsearch only.

7

failing to provide an else clause in the output is functionally equivalent to providing an else with a null-output:

 else {
  null {}
 }

The worry here, is it's sometimes tricky to get the logic "right", especially with negations and groupings in if-clauses, so at least while developing/prototyping, I tend to always include an else-clause to stdout until I'm happy that everything that is going to stdout is stuff I don't want in my other outputs.

Typically, when writing a complex pipeline, I add tags to events that explicitly match the patterns I'm looking for, and then use the presence of tags to determine which output(s) to send to; in your case, it looks like you have two separate outputs to handle the coercion of a missing [logType], so I would do something like the following:

filter {
  # ..

  # ensure we have a derived logType in our @metadata, even if the
  # original log message didn't include one.
  if ![logType] {
    mutate { "add_field" => {"[@metadata][logType]" => "error"} } 
  } else {
    mutate { "add_field" => {"[@metadata][logType]" => "%{[logType]}"} }
  }

  # ...

  # explicitly flag types of logs we do want in out output with the `export` tag:
  if [logType] != "access" {
    mutate { add_tag => "export" }
  }
  if [attribute][application] =~ /app1/ and [logType] != "redirect" {
    mutate { add_tag => "export" }
  }
}
output {
  if "export" in [tags] {
    amazon_es {
      hosts => ["search-qa1eslogs1-i6vqvywrmruuzpfm.us-west-1.es.amazonaws.com"]
      region => "us-west-1"
      index => "%{[attribute][ENVIRONMENT]}-%{[attribute][APPLICATION]}-%{[@metadata][logType]}-%{+YYYY.MM.dd}"
    }
  } else {
    # although the null-output isn't necessary, it may be helpful when combined
    # with the [X-Pack Pipeline Viewer][1], which shows consumption rates of
    # pipeline nodes in a web UI. 
    # [1]: https://www.elastic.co/guide/en/logstash/6.x/logstash-pipeline-viewer.html
    null {}
  }
}
8

@yaauie looks promising. I am going to change our configuration and inform you tomorrow.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.