# Filebeat multiline java app log4j logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 27, 2017, 6:34pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018 "2017-09-27T18:34:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ktella](https://avatars.discourse-cdn.com/v4/letter/k/71e660/32.png) [@ktella](https://discuss.elastic.co/u/ktella)\ **Post date:** [September 27, 2017, 6:34pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/1 "2017-09-27T18:34:20Z")
Need some help with filebeat multiline configuration of java App logs. I'm pasting sample logs from my app. For each request Ill have a Request and Response lines with some extra lines sometimes. All these logging is done by log4j. Logging lines are always not consecutive for a specific Request, logging lines from other threads could be printed in between. 09-27-2017 13:40:11,760 http-/10.98.88.26:8543-29 INFO com.xxx.xxxx - The username authenticate soap method of Token Authenticate Web Service was invoked Request1 09-27-2017 13:40:11,806 http-/10.98.88.26:8543-29 ERROR com.xxx.xxxx - Ping responded with Http Status code of [400] while performing username login... 09-27-2017 13:40:11,806 http-/10.98.88.26:8543-34 INFO com.xxx.xxxx - The username authenticate soap method of Token Authenticate Web Service was invoked Request2 09-27-2017 13:40:11,807 http-/10.98.88.26:8543-29 INFO [com.xxx.xxx](http://com.xxx.xxx) - The Token Authenticate Web Service Response Response1 09-27-2017 13:40:11,810 http-/10.98.88.26:8543-34 ERROR com.xxx.xxxx - Ping responded with Http Status code of [400] while performing username login... 09-27-2017 13:40:11,830 http-/10.98.88.26:8543-34 INFO [com.xxx.xxx](http://com.xxx.xxx) - The Token Authenticate Web Service Response Response2 ---
**Author:** ![ktella](https://avatars.discourse-cdn.com/v4/letter/k/71e660/32.png) [@ktella](https://discuss.elastic.co/u/ktella)\ **Post date:** [October 6, 2017, 4:48pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/2 "2017-10-06T16:48:50Z")
Any one ? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [October 6, 2017, 9:45pm UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/3 "2017-10-06T21:45:28Z")
> [@ktella](#): > > Logging lines are always not consecutive The multiline feature only works when the lines that need to be grouped are consecutive. I think the Logstash [aggregate filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html) may be applicable here. ---
**Author:** ![ktella](https://avatars.discourse-cdn.com/v4/letter/k/71e660/32.png) [@ktella](https://discuss.elastic.co/u/ktella)\ **Post date:** [October 7, 2017, 3:58am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/4 "2017-10-07T03:58:30Z")
Thank you. I kind of know what aggregate filter does. My current architecture is Filebeat(from multiple hosts) --\> KAFKA/Redis(Cluster) --\> Logstash(multiple instances) --\> ElasticSearch. Any idea how can this be implemented ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 8, 2017, 11:26am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/5 "2017-10-08T11:26:46Z")
You are looking for some kind of correlation/join operation. In order to pull this of you will need: - transaction 'ID' - parse/grok message - stable (hash-based) partitioning (as you have multiple instances) - (time windowed) join/correlation Re-consider if you really need to pre-corrlate the data in order to build your dashboards. Correlation can be 'expensive' and potentially bloats processing architecture. Also see: [Merging rails log lines](https://discuss.elastic.co/t/merging-rails-log-lines/102624/2) Btw. filebeat supports parsing json. Using log4j with [json layout](https://logging.apache.org/log4j/2.x/manual/layouts.html) and attaching an ID (e.g. via [ThreadContext](https://logging.apache.org/log4j/2.x/manual/thread-context.html) or custom field) one can configure the kafka output in filebeat to hash/partition based on the ID. In this case each Logstash instance will receive all events for a given ID -\> you can correlate via aggregate filter. Well, haven't used log4j for years, so no idea how easy it is to build something like this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 5, 2017, 11:26am UTC](https://discuss.elastic.co/t/filebeat-multiline-java-app-log4j-logs/102018/6 "2017-11-05T11:26:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.