Edit log before parsing?

Elastic Stack Beats
1

Hi,

I've got some logs that are badly formed JSON. I know the exact field with the problem (it's missing a value), but getting all the components that are doing this fixed is going to take some time.

Is there any way to fix this in the raw log line before it's parsed as a stopgap measure?

2

Hi @OffColour,

I am not sure if it can be solved from filebeat, it depends on the exact modification needed.

There are some options to modify already parsed log lines, you can for example add custom fields with the fields setting to add static custom fields to all events, you can also use processors to do some modifications, or define an ingest pipeline to do further modifications.

But if the line is not parsed because it is an invalid JSON, then this can be more complicated, you can try to collect the lines as normal log lines (without JSON parsing), and then try to modify them with the mentioned mechanisms and finally parse the JSON using the ingest processor for JSON fields.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.