Log4j input migration to Beats

Elastic Stack Logstash
1

Hello
As stated in the documentation log4j input plugin for logstash is deprecated (security reasons) and log4j2 input plugin (community) is quite not maintained...

The official recommendation for migration is to use filebeat.

For my use case (and probably not only mine) this is quite a setback:
log4j/log4j2 with MDC (mapped diag. context) and NDC (nested diag. context) which are used to pass additional information in the log event, beside the message and class context...

Currently log4j input plugin recognizes and automatically adds the MDC/NDC to the event in logstash, without any coding or specification

To partially do the same with beats, i will need to add %X{name} for every MDC entry to the log pattern, then parse that entry to a field in LS...
For the NDS its simpler, only %x once that can be simply parsed

My problem is that MDC is not a constant set, it can have lots of different key/value

Any sugestions?

2

Use a Log4j layout or whatever they're called that dumps the whole log record to JSON. Log4j 2 appears to include one and I'd expect it to support inclusion of MDC and NDC.

3

Thank you Magnus, did some digging and found some info about it:

Alternative 1 - Use JSON layout and write to file, then read with filebeat
for log4j2 - https://logging.apache.org/log4j/2.x/manual/layouts.html#JSONLayout
for log4j1 - https://stackoverflow.com/questions/29387007/does-log4j-support-json-format

  • + standard
  • - log4j1 does not natively support json layout, can use org.apache.log4j.xml.XMLLayout and parse XML
  • - this still creates a file to read with filebeat

Alternative 2 - use async appender with http appender
for log4j2 - https://logging.apache.org/log4j/2.x/manual/appenders.html#HttpAppender
for log4j1 - no standard appender

  • + no file
  • - volatile in memory buffer

Alternative 3 - use Kafka

  • - additional software

Alternative 4 - Write custom log4j appender for beats

  • - not written yet
  • - volatile in memory buffer

Will start testing Alternative 1

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.