How can Filebeat specify match rules to Logstash

Elastic Stack Logstash
1

I want to let Logstash'gork filter use the match rules which Filebeat give

Here is my Filebeat config:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /root/Log-test/test.log
  fields:
    "@metadata":
        formatter: "%{TIMESTAMP_ISO8601:timestamp} - %{NOTSPACE:module} - %{LOGLEVEL:level} - %{NOTSPACE:filename} - %{GREEDYDATA:log_message}"
  fields_under_root: true

output.logstash:
  hosts: ["localhost:5045"]

Here is my Logstash config:

input {
    beats {
        port => "5045"
    }
}

filter {
  grok {
    match => { "message" => "%{[@metadata][formatter]}" }
  }
}

output {
  file {
    path => "/tmp/log-test.log"
    codec => rubydebug { metadata => true }
  }
}

So, i want the grok know my match rules content (the message field) is "%{TIMESTAMP_ISO8601:timestamp} - %{NOTSPACE:module} - %{LOGLEVEL:level} - %{NOTSPACE:filename} - %{GREEDYDATA:log_message}"

But the setting above do not work, I want to know how can i implement the funciton like this? or is it possible to make it?

Thanks!

2

Hi @Gary.Pan,

I have not looked at Filebeat 7 very closely yet so I might be wrong but as far as I know, you can't really pass @metadata fields to Logstash that way... There are some @metadata fields from Filebeat that Logstash does get but they (at least used to be) are more or less predefined.

I also see no mention of @metadata fields in the Filebeat fields documentation.

I think I see what you are trying to accomplice. An interesting approach. Try adding just formater as a field and try

match => { "message" => [formatter] }

No idea if that will work or not though...

3

I have printed the data which Logstash get from Filebeat (you can see my Logstash output codec is rubydebug with metadata) and the data contain the formatter field, what is more i have use this way to pass my target index from Filebeat to Logstash. The problem of this case is not on the data transportation, is on how to express and assign the data to the message field, and i am sure the way you written match => { "message" => [formatter] } will raise an error, because i have tried:grinning:

4

I do not believe you can use a sprintf reference to a field in a grok filter. It interprets it as a pattern name reference, not a field name. So

    mutate { add_field => { "filename" => "foo.bar" "format" => "%{WORD:first}\.%{WORD:second}" } }
    grok { match => { "filename" => "%{format}" } }

just gets you an undefined pattern error for "format".

5

I use that way in Filebeat not in Logstash. Have you tried it in Filebeat?

6

No, I have not.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.