Custom field (filebeat) in condition in logstash filter

Elastic Stack Logstash
1

I have this config in filebeat.yml:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields_under_root: true
  fields:
    service_name: "nginx"
##### OutPut #######
output.logstash:
  hosts: ["x.x.x.x:5044"]

and in logstash config:

input {
   beats {
     port => 5044
   }
}
filter {
    if  [fields][service_name] == 'nginx' {
        json { source => "message" }
      }
}
output {
    if  [fields][service_name] == 'nginx' {
        elasticsearch {
        hosts => ["https://x.x.x.x:9200"]
        user => "user"
        password => "password"
        index => "logstash_nginx"
        cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
        ssl_certificate_verification => false
    }
        stdout{ codec => rubydebug }
  }
}

But nothing happens!? Where is my mistake? filebeat config or logstash config?
thanks.
******** Update *******
when I disable if in logstash config, filebeat sends the field correctly (service_name).
So, why logstash if not work?

thanks.

2

I think you to debug step by step

  1. Remove all the filter entries and see if you can see the data inside elasticsearch or stdout
  2. Check carefully for the field names, especially if the field name is. fields.service_name (which I don't think it is)
  3. Then finding the correct field name and put the if condition accordingly or try other fields to see

my experience, always shows test with no filters, then build it upwards, so you can fine tune.

3

@kelk thanks.
I disable if and look structure log in elstiacsearch and I see service_name is a root structure because I use fields_under_root: true. So I remove [fields] in logstash config. as a result

json-structure
json-structure1074×765 42.1 KB

filebeat.yml is:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  fields_under_root: true
  fields:
    service_name: "nginx"

and logstash config is:

input {
   beats {
     port => 5044
   }

}
filter {

    if  [service_name]  == "nginx" {
        json { source => "message" }
         mutate { remove_field => [ "@version", "message","log", "port", "tags","@timestamp", "input", "ecs", "[agent][ephemeral_id]", "[agent][id]","[agent][name]","[agent][type]", "[agent][version]" ] }
      }
}

output {

    if  [service_name]  == "nginx" {
        elasticsearch {
        hosts => ["https://x.x.x.x:9200"]
        user => "user"
        password => "password"
        index => "logstash_nginx"
        cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
        ssl_certificate_verification => false
    }
  }
5

_source is an elasticsearch construct, it does not exist in logstash.

if  [service_name]  == "nginx" {

would be correct provided that field_under_root is true.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.