Filebeat to logstash problem to parse json message

Elastic Stack Beats
1

Hello I am trying to parse json logs in filebeat and send to elastic without logstash
If a have message like in below works fine:

{"@timestamp":"2017-12-11T19:52:51.262-02:00","@version":1,"message":"foo","logger_name":"xpto","thread_name":"thread-1","level":"INFO","level_value":20000,"app_name":"foo-api","app_version":"2.4.5"}

But if a have json inside a message tag the message not parse

{"@timestamp":"2017-12-12T17:29:24.949-02:00","@version":1,"message":"{"signatu re": "foo", "args": "[foo [ bar: xxx ... "....

Have some way to parse json inside a message tag?

I am ussing filebeat 5.6.3
and configuration like below

paths:
- /var/log/foo/bar.log
document_type: json
json.keys_under_root: true
json.add_error_key: true
json.message_key: "message"

And I try to use decode_json_fields also but didint work

2

This is kind of use case that decode_json_fields is for so I would expect it to work if the embedded JSON is valid and properly escaped.

Can you provide a log sample that isn't truncated so that someone can try to replicate the issue?

Were there any errors/warnings in the Filebeat log?

3

Hello Andrew,

thanks for answer

I found in log debug

2017/12/13 21:23:13.770045 processor.go:67: DBG fail to apply processor decode_json_fields=message: invalid character '\n' in string literal
2017/12/13 21:23:13.771363 client.go:214: DBG Publish: {

Its possible to escape or remove this characters ?

4

There's no way to do this in Filebeat. You would need Logstash.

But shouldn't it be the responsibility of the thing creating these logs to write valid JSON?

5

Ok
This is a example of log I remove \n in this example

{"@timestamp":"2017-12-12T17:29:24.949-02:00","@version":1,"message":"{"signature": "aaa", "args": "123 | abc"}","logger_name":"br.com.ab.abcd.log.LoggingAspect","thread_name":"aaa-7-thread-1","level":"WARN","level_value":30000,"app_name":"ab-abcd-api","app_version":"2.4.5"}

but also not work

6

The contents of message are not properly escaped. The inner quotes should be escaped. See the linter output.

7

It works!

Thanks for pacience and help