Filebeat to logstash problem to parse json message

Hello I am trying to parse json logs in filebeat and send to elastic without logstash
If a have message like in below works fine:

{"@timestamp":"2017-12-11T19:52:51.262-02:00","@version":1,"message":"foo","logger_name":"xpto","thread_name":"thread-1","level":"INFO","level_value":20000,"app_name":"foo-api","app_version":"2.4.5"}

But if a have json inside a message tag the message not parse

{"@timestamp":"2017-12-12T17:29:24.949-02:00","@version":1,"message":"{"signatu re": "foo", "args": "[foo [ bar: xxx ... "....

Have some way to parse json inside a message tag?

I am ussing filebeat 5.6.3
and configuration like below

paths:
- /var/log/foo/bar.log
document_type: json
json.keys_under_root: true
json.add_error_key: true
json.message_key: "message"

And I try to use decode_json_fields also but didint work

This is kind of use case that decode_json_fields is for so I would expect it to work if the embedded JSON is valid and properly escaped.

Can you provide a log sample that isn't truncated so that someone can try to replicate the issue?

Were there any errors/warnings in the Filebeat log?

Hello Andrew,

thanks for answer

I found in log debug

2017/12/13 21:23:13.770045 processor.go:67: DBG fail to apply processor decode_json_fields=message: invalid character '\n' in string literal
2017/12/13 21:23:13.771363 client.go:214: DBG Publish: {

Its possible to escape or remove this characters ?

There's no way to do this in Filebeat. You would need Logstash.

But shouldn't it be the responsibility of the thing creating these logs to write valid JSON?

Ok
This is a example of log I remove \n in this example

{"@timestamp":"2017-12-12T17:29:24.949-02:00","@version":1,"message":"{"signature": "aaa", "args": "123 | abc"}","logger_name":"br.com.ab.abcd.log.LoggingAspect","thread_name":"aaa-7-thread-1","level":"WARN","level_value":30000,"app_name":"ab-abcd-api","app_version":"2.4.5"}

but also not work

The contents of message are not properly escaped. The inner quotes should be escaped. See the linter output.

It works!

Thanks for pacience and help

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.