Send metadata from filebeat to logstash

Elastic Stack Beats
1

Hi! I'd like to send a hostname and some other host metadata from filebeat to Logstash. Then Logstash saves files with a name like:

Summary 
file {
        path => "~/%{+YYYY-MM-dd}/%{component}/%{[@metadata][host]:[name]}-%{log_name}-%{+HH}.00.log.gz"
        codec => line { format => "%{message}"}
        file_mode => 0644
        gzip => true }

But Logstash can't parse this path, it saves a file like: ~/2019-08-06/app/%{[@metadata][host]:[name]}-weblogic-14.00.log.gz
I've tried a lot of different combinations of {, @,[ and " . Where am I wrong? Part of my Filebeat configuration:

Summary 
- type: log
  enabled: true
  paths:
     - /var/log/weblogic/weblogic.log
  fields:
    component: app
    env: test
    log_name: weblogic
  fields_under_root: true
processors:
  - add_host_metadata:
     cache.ttl: 5m

Also I tried - add_host_metadata: ~
What does tilde mean?
Of course, I can add a hostname in fields, but I'll have several paths and many servers so I don't want to enumerate each field in every path block.
Can anyone help, please!?

2

~ means null in YAML files. It means that add_host_metadata is running with the default configuration.

Could you please share your debug logs of Filebeat?

3

Thank you! Here you are: https://pastebin.com/4pTYg0Ei
This is journalctl output.
I changed some company's data like servername.

4

It was a bit hard to parse the output of journalctl, so it might not work. But I think you need to change the path to "~/%{+YYYY-MM-dd}/%{component}/%{[host]:[name]}-%{log_name}-%{+HH}.00.log.gz", as the event contains host.name on the top level of the fields.

1 Like
5

Yeah, it works! Thanks! :relieved:
But filebeat doesn't write log to file. I even added these configs:

Summary

logging.to_files: true
logging.level: debug
logging.files:
path: /var/log/filebeat
name: filebeat.log

But it doesn't matter now.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.