How can Filebeat send match rules to Logstash

[Gary.Pan (Nutto Pan)]

I want to let Logstash'gork filter use the match rules which Filebeat give

Here is my Filebeat config:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/Log-test/test.log
  fields:
    "@metadata":
        formatter: "%{TIMESTAMP_ISO8601:timestamp} - %{NOTSPACE:module} - %{LOGLEVEL:level} - %{NOTSPACE:filename} - %{GREEDYDATA:log_message}"
  fields_under_root: true

output.logstash:
  hosts: ["localhost:5045"]

Here is my Logstash config:

input {
    beats {
        port => "5045"
    }
}

filter {
  grok {
    match => { "message" => "%{[@metadata][formatter]}" }
  }
}

output {
  file {
    path => "/tmp/log-test.log"
    codec => rubydebug { metadata => true }
  }
}

So, i want the grok know my match rules content (the message field) is "%{TIMESTAMP_ISO8601:timestamp} - %{NOTSPACE:module} - %{LOGLEVEL:level} - %{NOTSPACE:filename} - %{GREEDYDATA:log_message}"

But the setting above do not work, I want to know how can i implement the funciton like this? or is it possible to make it?

Thanks!

[jsoriano (Jaime Soriano)]

Hi @Gary.Pan and welcome

This is not possible, because grok patterns are compiled on start time, so this field is not going to be read when events are being processed.

You should be able to parse this data by using an ingest node pipeline: https://www.elastic.co/guide/en/beats/filebeat/7.0/configuring-ingest-node.html

[Gary.Pan (Nutto Pan)]

Get it, thanks!