Dear fellow Elastic developers,
I am the maintainer of the log4j2-logstash-layout project, the fastest and the only fully customizable JSON layout plugin for Log4j 2: LogstashLayout. A couple of days ago (2019-08-30) Felix Barnsteiner from Elastic announced a new project: java-ecs-logging. This is great news for the Java world since the development gap between the code and getting your logs stashed to Elasticsearch is closing even more. java-ecs-logging transforms Log4j1/Log4j2/Logback LogEvent's into JSON strings compatible with the Elastic Common Schema (ECS). That said, I find the re-invention of yet another Log4j 2 JSON layout (EcsLayout) partly sad and disappointing for a couple of reasons:
LogstashLayout is already a battle-tested well-established solution used by hundreds of companies around the world.
LogstashLayout has been enhanced with many feature requests/PRs provided by its users. Its design and existence does not emanate from hypothetical uses cases.
LogstashLayout supports way more features and allows full schema customization where none is available in EcsLayout.
One can perfectly support ECS in LogstashLayout by just changing the JSON schema, no plugin/software update is needed.
Now there are 3 contenders in the market: JSONLayout, LogstashLayout, and EcsLayout.
I have briefly benchmarked EcsLayout and LogstashLayout and observed that EcsLayout is ~0.5X faster for lite LogEvents and LogstashLayout is faster ~10X faster for full (stack trace, MDC, NDC, etc.) LogEvents. In each case LogstashLayout showed the lowest GC load. That said, do you really need something faster given LogstashLayout can render 793,597 LogEvent/sec on a single core? (I will share my findings once java-ecs-logging publishes an artifact to Maven Central.)
EcsLayout doesn't have any dependencies. LogstashLayout depends on Jackson, though provides a fat JAR artifact along with every release.
I would like to kindly ask you to consider a unifying approach rather than deprecating existing solutions with inferior new alternatives. Rather than coming up with a new Log4j 2 layout, IMHO, java-ecs-logging could have chosen (and can still chose!) one of the following paths:
- Contributing to LogstashLayout for the missing features, if there is any
- Using LogstashLayout within the project
- Relaying the Log4j 2 user base to LogstashLayout
I would really appreciate a solution that would benefit both Elastic and its existing Log4j 2 users. I am open to any kind of collaboration along this direction and will be looking forward for your response.