Exporting Only Part of a Message

[seth.yes (Seth S)]

I am looking to use Filebeat 6.5 to only export only the json of a log message. Current unrelated issues disallow me from modifying the way the raw message is formatted, nor the way the data is processed or indexed once off a host. For the following example code, I'd like Filebeat to only export the json -- if it's easier, the payload could be considered anything after the : before the json.

Jan 19 21:02:39 testbox1-aws my-service[94636]: {"level":"info","ts":1547931759.8724036,"caller":"agent/runner.go:172","message":"my message and other keys may be stored in this hash","service_name":"my-service","zone":"my-zone"}

Therefore my message to be sent to Logstash/Elasticsearch would be:

{"level":"info","ts":1547931759.8724036,"caller":"agent/runner.go:172","message":"my message and other keys may be stored in this hash","service_name":"my-service","zone":"my-zone"}

Does Filebeat support this? I have been looking at decode_json_fields, however that appears to expect the entire payload to be json..

[Magnus_Kessler]

Your best option here is to extract the message inside a logstash (grok) filter, and then use the JSON part with the json filter plugin. If you don't want to use Logstash, the same can be achieved with Elasticsearch ingest processors, too.

[seth.yes (Seth S)]

Yeah, I don't currently have access to any of the downstream processing, thus why I was hoping to process the message on the host machine prior to exporting. Thanks for the suggestion!