Unifying Log4j 2 layouts and EcsLayout

Elastic Observability Logs
1

Dear fellow Elastic developers,

I am the maintainer of the log4j2-logstash-layout project, the fastest and the only fully customizable JSON layout plugin for Log4j 2: LogstashLayout. A couple of days ago (2019-08-30) Felix Barnsteiner from Elastic announced a new project: java-ecs-logging. This is great news for the Java world since the development gap between the code and getting your logs stashed to Elasticsearch is closing even more. java-ecs-logging transforms Log4j1/Log4j2/Logback LogEvent's into JSON strings compatible with the Elastic Common Schema (ECS). That said, I find the re-invention of yet another Log4j 2 JSON layout (EcsLayout) partly sad and disappointing for a couple of reasons:

  • LogstashLayout is already a battle-tested well-established solution used by hundreds of companies around the world.

  • LogstashLayout has been enhanced with many feature requests/PRs provided by its users. Its design and existence does not emanate from hypothetical uses cases.

  • LogstashLayout supports way more features and allows full schema customization where none is available in EcsLayout.

  • One can perfectly support ECS in LogstashLayout by just changing the JSON schema, no plugin/software update is needed.

  • Now there are 3 contenders in the market: JSONLayout, LogstashLayout, and EcsLayout.

  • I have briefly benchmarked EcsLayout and LogstashLayout and observed that EcsLayout is ~0.5X faster for lite LogEvents and LogstashLayout is faster ~10X faster for full (stack trace, MDC, NDC, etc.) LogEvents. In each case LogstashLayout showed the lowest GC load. That said, do you really need something faster given LogstashLayout can render 793,597 LogEvent/sec on a single core? (I will share my findings once java-ecs-logging publishes an artifact to Maven Central.)

  • EcsLayout doesn't have any dependencies. LogstashLayout depends on Jackson, though provides a fat JAR artifact along with every release.

I would like to kindly ask you to consider a unifying approach rather than deprecating existing solutions with inferior new alternatives. Rather than coming up with a new Log4j 2 layout, IMHO, java-ecs-logging could have chosen (and can still chose!) one of the following paths:

  • Contributing to LogstashLayout for the missing features, if there is any
  • Using LogstashLayout within the project
  • Relaying the Log4j 2 user base to LogstashLayout

I would really appreciate a solution that would benefit both Elastic and its existing Log4j 2 users. I am open to any kind of collaboration along this direction and will be looking forward for your response.

Best.

1 Like
2

Disclaimer: @felixbarny s on vacation this week. He'll be able to explain things much better and in more detail, but I'll get started so it doesn't look like we're ignoring you :slight_smile:

First, thanks a lot for the ongoing work on log4j2-logstash-layout. A strong open source ecosystem is a major part in the success of the stack and we don't have any intentions of changing that. I don't think we have been "deprecating existing solutions" — java-ecs-logging is an approach to solving a different problem:

  • Equal support for Log4j1, Log4j2, and Logback — independent of your logging backend we want to provide a similar solution.
  • Opinionated support for ECS — this doesn't have all the features, but should be as simple to use as possible. There's one path from your log appender over Filebeat to Elasticsearch in ECS that allows for the correlation of traces, works well in the Logs UI,...

This isn't competition to log4j2-logstash-layout or logstash-logback-encoder. If you need more features and want to go "deeper" with one specific log appender, please use the existing projects — we're keeping ours intentionally simple. I'll leave it up to @felixbarny, but it might make sense to add a section to the readme: If you need additional features, take a look at ...

And I'm sure Felix will also take a look at the performance part. I'm not sure what we have optimized for or benchmark, but thanks for the pointer!

For the missing features I guess that would be an out of the box ECS compatible format for LogstashLayout. I guess we could provide that pretty easily.

3

Thanks for the detailed reply @xeraa, appreciated.

I do get your points, but I am having difficulty in seeing they address my concerns. Let me try to express my question more tangibly:

Everything that is doable by EcsLayout of java-ecs-logging (and way more!) is possible via the following LogstashLayout configuration:

<LogstashLayout
  timeZoneId="UTC"
  dateTimeFormatPattern="yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
  eventTemplate="{\"@timestamp\":\"${json:timestamp}\",\"log.level\":\"${json:level}\",...,\"labels\":\"${json:mdc}\",\"tags\":\"${json:ndc}\"}"
  stackTraceEnabled="true"/>

EcsLayout is not doing anything more than this. On the contrary, EcsLayout constrains you with a certain schema and necessitates artifact upgrades for ECS changes. Whereas LogstashLayout doesn't put any constraints and can easily adapt a new ECS by just changing the eventTemplate property.

  • You wanted equal support for Log4j1, Log4j2, Logback. That is a valid wish for long-term maintainability. But given there exists pretty well-established readily-used solutions, why reinventing the wheel with an inferior one? Further you can very well reach this goal by separate artifacts. This wish doesn't necessarily mean one umbrella project to host layouts for all existing loggers. Are you worried about how to shepherd log4j2-logstash-layout in the long term? Fine. Let's move it to github.com/elastic group. It has an Apache License v2.0. As long as me and the rest of the world can contribute, I am ok with the move. At the end of the day, it is a piece of software serving to Elastic.

  • Opinionated ECS support? It is just one JSON schema that you need to provide to LogstashLayout configuration. Your opinionated ECS support is supported.

  • Do you see any particular features missing in LogstashLayout compared to EcsLayout? Tell me, I am all ears. I will do my best to get them in, unless you want to contribute them yourselves.

Let me repeat, I really appreciate your thoughts. I am not intending to drive the discussion to a wall. Rather I am trying avoid segmentation that would result in deprecation of log4j2-logtash-layout along with its rich feature set and noteworthy user base.

(I will be looking forward for @felixbarny's reply after he is back from vacation.)

2 Likes
4

I think the configuration example nicely shows the difference in the approaches: Yes, you can add this, any custom configuration, and way more. But what if you don't need and want all that power (and complexity)? That constraint is a feature if you don't need to wrestle any configurations or look through a list of features — it just does the right thing for one specific use-case.

And that EcsLayout is part of the artifact is IMO a feature. If you maintain that configuration by hand you'll need to check every release of ECS for additions or breaking changes (these are and will be happening). If the ECS mapping is part of the dependency that is happening automatically for you. With the right release notes and something like Dependabot you'll immediately see if that upgrade works for your version of the Elastic Stack or not.

It's different use-cases being solved and depending on what you have at hand you will be able to pick the one that matches your requirements best. I don't see one as the replacement for the other and I don't assume we want to take over LogstashLayout — it's great that you (from the community) are maintaining this with a different focus than what we are doing.

5

First of all, I want to say to @vyazici, that your work is great and I am fully support your concerns.

The lack of logstash-layout for log4j2 has been a nuisance for a couple of years. The last commit in official repo was 5 years ago. Heck, there is barely any reaction from the maintainers and no deprecation notice in the docs. This is very disappointing, and the position of Elastic team is at least strange for me.
My team ended up making our layout, and we are not the only ones.

@xeraa I think your argument about configuration is invalid. @vyazici pointed out that his solution is superior in many ways and demonstrated that EcsLayout can be implemented with his solution. Pointing that is hard to configure, I think, is a different question and not a problem. For example, @vyazici's solution can be modified to support EcsLayout as native one and be enabled with a simple flag in config. To be in sync with schema updates is organizational problem, not a programming one.

Also, restricting your users to configure something without gaining anything is a nightmare. The users of logstash-layout are mostly devOps or programmers with tons of use-cases. The unconfigurable solution is an inferior one for them. Heck, we are talking about OSS - freedom is one of the main pillars of it. Yes, easy-of-use is an important feature, but I think its totally solvable in @vyazici's solution and can be improved.

@xeraa To be honest, your response sounds a lot like "Not invented here" syndrome. This is cancer. And still @vyazici proposed solution even for that.

6

To me it seems that java-ecs-logging and log4j2-logtash-layout solve different problem sets and can happily coexist. We don't have the intention of deprecating other solutions. But we created ECS to make it easier to correlate between different data sets. All our solutions, like the Logs UI, SIEM, APM adhere to ECS which makes the integration with them much more pleasant. For anyone not wanting to jump on ECS, they can use your great log4j2-logstash-layout library :slight_smile:

The java-ecs-logging library intentionally only tries to solve a more narrow use case. Having a narrow problem set usually makes it easier to create something which is easy to use, aggressively optimized and less complex.

java-ecs-logging is implemented in less than 1000 lines of code and does not come with any external dependencies which can make dependency conflicts more likely. Also, not everyone is comfortable pulling in Jackson. There are quite a few CVEs filed for it.

What are the use cases you are missing in java-ecs-logging? I'd like to hear more about it :slight_smile:

I think it's great to have a library like log4j2-logtash-layout which provides generic JSON support. But if you want to leverage ECS and the Logs UI, you don't need that flexibility. That's where java-ecs-logging comes in.

I also ran some benchmarks and saw quite different results:

OpenJDK 11 MacBook Pro 15-inch, 2017

Benchmark ops/sec* MB/sec*
liteEcsLayout 1,532,428 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ (100%) 0.0
fullEcsLayout 39,154 ▉ (3%) 1,157.9
liteLogstashLayout 30,194 ▉ (2%) 14,862.9
fullLogstashLayout 21,323 ▉ (1%) 10,502.0

It seems that since #27 Ditch off thread locals which are causing more harm than benefit. · vy/log4j2-logstash-layout@2a11e7f · GitHub (Ditch off thread locals which are causing more harm than benefit.), the performance degraded quite a lot, at least on the setup I have tested it on (also tested with OpenJDK 9).

When comparing the allocation rate, it's better to look at the allocated bytes per operation as opposed to MB/sec. Otherwise, you penalize the faster solution with more ops/sec.

Here are some benchmark results with v0.17 of log4j2-logtash-layout, which is before the Ditch off thread locals which are causing more harm than benefit commit.

LogstashLayoutBenchmark.fullEcsLayout                                    thrpt    6        27.476 ±      0.602   ops/s
LogstashLayoutBenchmark.fullEcsLayout:·gc.alloc.rate.norm                thrpt    6  31527207.447 ±      0.592    B/op
LogstashLayoutBenchmark.fullLogstashLayout                               thrpt    6        79.421 ±      4.955   ops/s
LogstashLayoutBenchmark.fullLogstashLayout:·gc.alloc.rate.norm           thrpt    6    104002.594 ±      0.176    B/op
LogstashLayoutBenchmark.liteEcsLayout                                    thrpt    6      1493.281 ±     22.187   ops/s
LogstashLayoutBenchmark.liteEcsLayout:·gc.alloc.rate.norm                thrpt    6         0.138 ±      0.013    B/op
LogstashLayoutBenchmark.liteLogstashLayout                               thrpt    6      1096.723 ±     23.821   ops/s
LogstashLayoutBenchmark.liteLogstashLayout:·gc.alloc.rate.norm           thrpt    6         0.186 ±      0.004    B/op

And another benchmark using the default LogstashJsonEventLayoutV1.json:

LogstashLayoutBenchmark.fullEcsLayout                                    thrpt    6        28.005 ±       0.757   ops/s
LogstashLayoutBenchmark.fullEcsLayout:·gc.alloc.rate.norm                thrpt    6  31527207.222 ±       0.196    B/op
LogstashLayoutBenchmark.fullLogstashLayout                               thrpt    6        40.841 ±       9.820   ops/s
LogstashLayoutBenchmark.fullLogstashLayout:·gc.alloc.rate.norm           thrpt    6  63359204.985 ±       1.179    B/op

I'm not convinced it makes sense to create one library to rule them all. Sometimes it is better to have different libraries targeting different use cases. To me it seems that is the situation here. But I'm definitely interested in hearing other opinions and in collaborating :slight_smile:

7

@felixbarny, see the most recent log4j2-logtash-layout benchmark:

20191004-logtash-layout-benchmark
20191004-logtash-layout-benchmark.png804×424 40.9 KB

I would appreciate if you can check the way I have benchmarked LogstashLayout against EcsLayout. In README.md, I have also added a feature comparison matrix. Last, I have added EcsLayout template to the artifact too. Hence people can still use LogstashLayout for ECS-compatible logging.

If you would approve that I have configured EcsLayout fairly, I will make a new release.

8

Hi again :slight_smile:

Congrats to your latest optimizations, looking great!

Some feedback regarding the benchmarks: I’d disable the stack trace profiler as it causes allocations. Another ask would be to also publish results of multi-threaded benchmarks. This would probably be a more realistic scenario and it is one where EcsLayout shines as it does not use any locking or synchronization. Otherwise, the setup looks good. The additionalFields new KeyValuePair[0] is not necessary but does not harm either (it does not cause a NPE in 0.1.1).

Regarding the feature comparison matrix, there are some things you might want add. But I’m not really sure how exhaustive that list is indented to be so feel free to ignore.

  • I’ve added support for serializing ObjectMessages with Jackson if Jackson is on the class path (it’s an optional dependency). So that probably ticks the “Custom typed Message serialization?” box. See https://github.com/elastic/java-ecs-logging/pull/37
  • Java compatibility (EcsLayout supports Java 6)
  • Rendering stack traces as multi-line JSON array (readable JSON stack traces)
  • topLevelLabels (enables APM/Log correlation)

Thanks for adding a built-in template for EcsLayout. I’ve noticed that EcsLayout.json contains a hard-coded value for service.name though. You might want to remove that and only use the hard-coded version in src/test/resources for the benchmarks.

1 Like
9

Done.

I am not promising for that, but I will see what I can do.

Removed.

I think it should be exhaustive enough to let users make a well-informed decision while choosing. So if you have any more thoughts, keep them coming.

Done.

Done.

Done.

Isn't this already covered by supported additional fields of all 3?

Removed.

Thanks for your kind help. I will add your name to the list of contributors and make a new release with these changes.

2 Likes
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.