Logstash - selectively applying filter

[rs_ela]

Hi there

I am sending logs from my nginx instance to ELK via Filebeat. The problem is that nginx access logs are json, but error logs are not and Logstash reports a lot of parsing errors which then swamp the log file on the disk. In order to avoid that I temporarily upped Logstash log level to critical, but I am not happy with that solution.

The goal:

• accept both error and access logs from nginx
• avoid getting parsing errors
• nginx access logs are in json, must be
• nginx error logs are plaintext and it's fine (but could be json as well, no matter)

Solutions?

1. configure nginx so that both access and error logs are json - but I believe it's currently not possible with reasonable amount of work
2. configure Logstash so that it uses json codec on access logs and plaintext codec on error logs - but how do I do that?

My current Logstash config is as below. Tried configuring Logstash according to the 2nd solution, but failed. It seems that someone has already faced such issue (Can I use both json and plain logs in a common filebeat input?), but I am not sure how do I "selectively apply a json filter if the line looks like JSON"?

Could someone help? Many thanks in advance for any suggestions.

input {
    beats {
        port => 1234
        codec => "json"
        }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
    }
}

[magnusbaeck (Magnus Bäck)]

Can't you do the JSON parsing on the Filebeat side instead? Or at least assign a distinct type (with document_type) to each kind of log so that you can conditionally apply a json filter on the Logstash side?

[rs_ela]

I would like to use as little computing power on the nginx side as possible. So even if json parsing does not seem like a cpu-hungry operation I'd rather stick with option no 2.

So this would be config on Filebeat side:

- input_type: log
  paths:
    - /var/log/nginx/*/access.log
  document_type: metricsets

- input_type: log
  paths:
    - /var/log/nginx/*/error.log
  document_type: nginx_error_log

Then how do I approach conditionally appling json filter in Logstash input config?

[magnusbaeck (Magnus Bäck)]

Then how do I approach conditionally appling json filter in Logstash input config?

Accessing event data and fields | Logstash Reference [8.11] | Elastic

[rs_ela]

So we've got something like this:

input {
    beats {
        port => 1234
        }
}

output
{
        if [type] == "nginx_error_log"
        {
            elasticsearch
                {
                codec => plain
                hosts => ["localhost:9200"]
                }
        }
        else
        {
             elasticsearch
                {
                codec => json
                hosts => ["localhost:9200"]
                }
        }
}

Thought that might do the job, but by the time the data reaches output section, the codec (plaintext) has already been applied by the beats input plugin.

Should I apply the filter in the input section instead?

Or maybe should I try and mutate the codec in the output section? Don't know if this idea is a good one?

[magnusbaeck (Magnus Bäck)]

Don't set the codec option for the elasticsearch output. Use a json filter instead and wrap it in a conditional.

[rs_ela]

Right. Managed to come up with working config.

Filebeat:

- input_type: log
  paths:
    - /var/log/nginx/*/access.log
  document_type: nginx_access_log

- input_type: log
  paths:
    - /var/log/nginx/*/error.log
  document_type: metricsets

Logstash:

input
{
        beats
        {
                port => 1234
        }
}

filter
{
        if [type] == "nginx_access_log"
        {
                json
                {
                        source => "message"
                }
        }
}

output
{
        elasticsearch
        {
                hosts => ["localhost:9200"]
        }
}

Will let it work for a couple of days and see what happens, but so far it's delivering. Thank you for your help!