Filebeat source field missing in 7.8

Elastic Stack Beats
1

We've recently upgraded our elastic stack and have upgraded our Filebeat version from 5.6.14 to 7.8.1. The old version used to provide a "source" field that contained the path name of the log file. This field is missing in the new version and i don't see any field in the output that has the log file path name in it. Is there some way I can configure Filebeat 7.8.1 to provide this field. I can't find this in the documentation. Thanks.

2

@PMDubuc A bit unsure which one you mean, if you mean the specific logfile that is currently being processed by filebeat, it should be under log.* fields.

Do you use this with modules, or is this a custom one using filebeat.yml?

Do you have any custom input processors or ingest pipelines that might remove fields?

3

I'm not using modules. I use inputs instead. I don't see the log.* fields. I think I know what's happening. The log.* fields are getting overwritten by a log field in log messages that are formatted in JSON. Is there any way in Filebeat to rename the log.* fields to something like filebeat-log.* so they aren't overwritten when these JSON logs are parsed in the Logstash filter?

4

I solved this by renaming the log field at the top of my logstash filter:

    mutate {
        rename => { "log" => "filebeat-log"}
    }

so the log.* fields become filebeat-log.* and they don't get overwritten by messages with that field name in them.

5

Happy to hear you got it resolved! :slight_smile: Another option if you like to resolve it on the filebeat side is to use processors, we have one for renaming for example.

6

Thanks! Since we have fewer instances of Logstash than Filebeat this way will be easier for us.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.