Filebeat and Magento2 reports

[sergiu.tot]

Hello guys,

I have a Magento2 instance for which I collect the logs using prospectors in filebeat-5.2.1. This is happening on a docker container based on ubuntu xenial (16.04).

The Magento2 report is a valid json file with a single line, but contain no newline at the end of the string which results in logs not being sent - https://www.elastic.co/guide/en/beats/filebeat/current/faq.html#newline-character-required-eof

Tried with single/multi line, tried changing the close_timeout and a few other suggestions found online but nothing works so far.

Could you provide some suggestions on how to approach it?

[steffens (Steffen Siering)]

Problem is, lines are split on newline. The json parsing and multiline filters are applied thereafter.

Given JSON is structured, I would like to beats to not requiring the newline, but just stream through the JSON document, parsing object after object after object. Without requiring newlines, but also supporting pretty printed JSON. This is unfortunately not possible as of now. But we're well aware of this use-case. For this and other common use-cases I would like the filebeat readers to be somewhat chainable, so we can plugin decompression, json, xml, multiline, newline splitting, content splitting on custom pattern in any order required.

Unfortunately I didn't find a ticket, but feel free to open an enhancement request.

[sergiu.tot]

Hello Steffen,

Thanks for your answer. I just opened an enhancement request: https://github.com/elastic/beats/issues/6132