# Parsing Log4j2 logs from filebeat based on defined Pattern Layout **URL:** \ **Category:** Logstash\ **Created:** [September 26, 2018, 7:11am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941 "2018-09-26T07:11:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![sbienert](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Post date:** [September 26, 2018, 7:11am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/1 "2018-09-26T07:11:44Z")
Hello, I want to parse my logs from Filebeat into Elasticsearch as if there would be an Elasticsearch appender in Log4j2.xml meaning correct fields with their values and not one field with message where every field values are put in as a string. I already read that probably a grok filter in Logstash would be the way to go. However, I am wondering if there is a method to get a grok filter config based on the defined Log4j2 pattern layout. If not wouldn't this be possible to do and maybe a good idea to somehow extend the product? Or maybe for some people to write a plugin. Can you help me with a grok filter for this pattern Layout?: %d{yyyy-MM-dd HH:mm:ss.SSS} ${LOG\_LEVEL\_PATTERN:-%5p} ${PID:- } --- [%t] %-40.40logger{39} : %m%n${LOG\_EXCEPTION\_CONVERSION\_WORD:-%wEx} Thank you very much ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 26, 2018, 7:16am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/2 "2018-09-26T07:16:04Z")
Why not just a layout that produces JSON? Then you don't have to do any parsing. ---
**Author:** ![sbienert](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Post date:** [September 26, 2018, 7:33am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/3 "2018-09-26T07:33:23Z")
I cannot change the layout. These logs are from products that we also deliver to our customers. So changing the log4j2.xml is not a real option and thats also why I would not like to add a rest appender since I would always have to add this appender and restart the app when we install updated versions on our systems. ---
**Author:** ![sbienert](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Post date:** [September 26, 2018, 2:32pm UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/4 "2018-09-26T14:32:15Z")
I think I am good doing this with grok by myself so please hav this on hold. ---
**Author:** ![yirduhudro](https://avatars.discourse-cdn.com/v4/letter/y/7c8e57/32.png) [@yirduhudro](https://discuss.elastic.co/u/yirduhudro)\ **Post date:** [September 27, 2018, 12:29pm UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/5 "2018-09-27T12:29:40Z")
For [vidmate](https://vidmate.onl/) me the [tutuapp](https://tutuappx.com/) gork is working [plex](https://plex.software/) fine so far. Regards, Smith ---
**Author:** ![sbienert](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Post date:** [September 28, 2018, 10:10am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/6 "2018-09-28T10:10:48Z")
Hello @magnusbaeck, Grok works fine for me so far, but some log files have a different layout than others with no specific pattern. My idea was to create a grok filter for all existing patterns until one pattern does not throw a \_grokparsefailure anymore. For this I would have to check the number of \_grokparsefailures in the [tags] with an if condition and go through all patterns until there is one \_grokparsefailure too little. You have an idea how to this? Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 28, 2018, 10:37am UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/7 "2018-09-28T10:37:29Z")
I don't understand. There can't be more than one `_grokparsefailure` tag. The `tags` field is essentially a set, i.e. there can't be duplicates. Why not just use a single grok filter that lists all the grok expressions you think you need and then take care of all messages that fall through and result in a `_grokparsefailure` tag? ---
**Author:** ![sbienert](https://avatars.discourse-cdn.com/v4/letter/s/a3d4f5/32.png) [@sbienert](https://discuss.elastic.co/u/sbienert)\ **Post date:** [September 28, 2018, 1:33pm UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/8 "2018-09-28T13:33:57Z")
I did not know that if one match does not result in a grokparsefailure, it skips the others and when it does it goes to the next one. This made it for me. Thanks for your help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 26, 2018, 1:34pm UTC](https://discuss.elastic.co/t/parsing-log4j2-logs-from-filebeat-based-on-defined-pattern-layout/149941/9 "2018-10-26T13:34:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.