Json + multiline codecs together, is it possible?

[Hleb_V]

Hi.

We are running Java applications inside docker containers. We use file input with json codec to parse their logs. Unfortunately Java is known to throw stack traces and docker creates a single json entry for every log line. With plain files we can use multiline codec to join them into a single event but how to use it with json? It looks like filebeat can be used instead but version 6.8 we have to use now doesn't support other required features, so is it possible to use just logstash in this scenario?

[Badger]

You cannot have multiple codecs on a single input. I suggest you use a multiline codec and then a json filter.

[Hleb_V]

Hi Badger,

could you share example of input section for such case?

[Badger]

No, I have no idea what your logs look like, so I cannot guess what configuration you would need to the multiline filter.

[Hleb_V]

docker logs are typical: {"log":"log entry begins here (no leading space)","stream":"stdout","time":"iso timestamp"} and {"log":" and continues here (there is a leading space)","stream":"stdout","time":"iso timestamp"}.

If I understand correctly multiline codec accepts raw text, so I don't understand how to make it work with json.

[GreenEyed (Daniel López)]

One option would be to send the logs directly to logstash from the application (we use logstash-gelf for that) and then you don't even need to install filebeat in the machines.
We also send the regular text logs to console so the output is there if the logstash connection fails and someone needs to look at them, but we then rotate them quite aggresively.