Logstash not able to access the custom field from Filebeat

[Dattatray_Kulkarni]

Hi,

I have multiple types of log files from different servers. So I am using multiple filebeat agents.

In the Filebeat configuration, I am adding a custom field - log_type
Here is the snippet of the filebeat configuration.

fields:
log_type: threadpoolworkerlog

Filebeat is producing the correct output and sending it to Logstash.

In Logstash, based on the value of this field log_type, I want to add some filters.
Here is the relevant section of the Logstash config.

filter {
if [fields][log_type] == "threadpoolworkerlog" {
grok {
add_tag => [ "threadpoolworkerlog" ]
add_field => { "log_type" => "threadpoolworkerlog" }
match => { "message" => "%{TIMESTAMP_ISO8601:datetime} %{DATA:thread} %{LOGLEVEL:Loglevel} %{DATA:Classname} %{GREEDYDATA:messagetext}"}
}
}
}

But logstash is not reading the value of the custom field log_type, so it is not able to execute the statements in the grok block - add tag, add_field, etc.

Is there anything incorrect in the logstash configuration?

[batrako (Iván José Alvarez Domínguez)]

@Dattatray_Kulkarni

fields][log_type]

in my case with 5.4 i use this option in filebeat :

fields_under_root: true

this allow you use filebeat fields at toplevel in logstash.
With this, in logstash, i can use custom filebeat fields as common variables. For example:
index => "myindex-%{[entorno]}-%{+YYYY.MM.dd}" where [entorno] is my custom field in filebeat.