Filebeat journald input truncates custom fields at ~64KiB

I have an issue where custom fields in systemd-journald entries are being truncated at ~64KiB. The regular "MESSAGE" field doesn't seem to be affected, only custom fields.

I can't seem to find an option in the docs that might allow for longer fields, am I missing something obvious?

To reproduce:

# filebeat.yml
filebeat.inputs:

- type: journald
  include_matches.match:
    - SYSLOG_IDENTIFIER=test

output.file:
  path: "/tmp/filebeat"
  filename: filebeat
// Create a journal event with a custom field called "data", with 89784 chars of text
const Journald = require('systemd-journald');
const logger = new Journald({syslog_identifier: 'test'});
const data = 'abcdefghijklmnopqrstuvwxyz...'.repeat(3096)
logger.info('Test message', {"data": data}) 

Verify that the length is correct in systemd-journald (aside from an added trailing newline)

journalctl MESSAGE="Test message" -o json --all --no-pager|jq -r .DATA | wc -m
> 89785

Look at the filebeat output, see that it has been truncated.

cat /tmp/filebeat/filebeat-20240207.ndjson | jq -r .journald.custom.data | wc -m
> 65532

If I enable debug logging in Filebeat I see the truncated message in a processing/processors.go "Publish event" log message, so I suspect it's an input issue, not an output issue. In production I use a logstash output and have the same problem.

Any suggestions how I might get round this limit?

Optimistically giving this a bump. I've started digging around the source, but my go-fu is weak and I don't immediately see any obvious issues.

The truncation of custom fields in systemd-journald entries at around 64KiB by Filebeat is likely due to internal limitations or the way Filebeat processes these fields, as there isn't a documented option to handle larger fields directly. Consider these workarounds:

  1. Split large data into multiple smaller fields or logs.
  2. Use a custom script to preprocess and forward logs.
  3. Deploy Logstash as an intermediary for more flexible data handling.
  4. Provide feedback to Elastic to potentially address this in future updates.
  5. Ensure you're using the latest version of Filebeat for any recent fixes or improvements.

Direct configuration to extend field sizes isn't currently available, so adapting your approach is necessary.