Filebeat - Single-line JSON log ingestion: expected behavior and required configuration

Elastic Stack Beats
1

Hi,

I have a specific use case and I need to know if Filebeat can handle it.

My log files consist of a single very long line of JSON (no newline characters),
which can be up to 27 MB in size. The entire file content is one single JSON object
or array, all on one line.

Example (simplified):
{"events":[{"timestamp":"...","level":"INFO","msg":"..."},{"timestamp":"...","level":"ERROR","msg":"..."},...]}

Questions:

  1. Can Filebeat ingest a file where the entire content is a single line of ~27 MB?
    I am aware of the default max_bytes limit - is raising it enough, or are there
    other hard limits to be aware of?

  2. Is there a recommended approach to handle this kind of payload in Filebeat
    (multiline config, custom parsing, etc.)?

  3. Would it be better to pre-process the file before Filebeat picks it up
    (e.g. split into multiple lines), or can Filebeat handle this natively?

  4. Are there known performance or memory concerns when processing single lines
    this large?

Filebeat version: 9.3.2
OS: Windows / Linux

Thanks in advance!

2

Hello and welcome,

Filebeat is not the right tool for this because a couple of issues.

Filebeat requires lines to end in a newline character, it will not read a line if there is no newline.

So the expected behaviour is that filebeat will not read your file.

If you have an array of json objects and want each item to be a document in your destination, then you would need to split the items into individual documents, but Filebeat cannot do that.

In your case, Logstash can do both, it can read an entire file without waiting for a newline character and it can split an array field into multiple events.

3

Hello,

Thank you for the clear explanation that makes a lot of sense.

I understand now why Filebeat isn't suitable here: since my log files consist of a single long JSON line with no newline character, Filebeat simply won't read them. And even if it could, splitting a JSON array into individual documents is beyond its capabilities.

I'll switch to Logstash as you suggested, since it can handle both reading the full file and splitting the array into separate events.

Thanks again for pointing me in the right direction.

Best regards