Keeping `message` field intact with module parsing

Elastic Stack Beats
1

Parsing the message field with filebeat processors is great, as it allows structured search for logs, but one thing that surprises me is that the message field itself is destroyed. This means that just casual viewing of logs in Kibana or tools like elktail will just shown an empty log message. Here is an example from a Kibana dashboard:

image
image.png1146×426 18.6 KB

You then have to either add several of the destructured fields to the output, or click into the details of each one. This is not appealing when just trying to get an overall view of a set of logs before digging into the details.

In addition, if I search for message:(something) I won't find it. I have to know which destructured field contains something to do a search.

Can filebeat be configured to parse data out of the message, but leave the message field as-is rather than destroying it?

2

Which processor are you referring to? Processors add_locale, add_*_metadata, include_fields only enrich events with additional data, drop_* drops parts of the events or the whole event, targets of decode_json_fields and dissect are configurable.

3

@kvch Sorry, I didn't specify: this is with filebeats running inside Kubernetes, with auto-discover enabled and hints on. The example logs I gave a screenshot above was with a pod annotated with:

co.elastic.logs/module: apache2
co.elastic.logs/fileset: access
4

Sorry, I misunderstood. I thought you were referring to the processors of Filebeat, not the modules.
Right now it is not supported. Do you mind opening an enhancement request? https://github.com/elastic/beats/issues/new

5

Done: Keep `message` field intact when using modules · Issue #8950 · elastic/beats · GitHub