# Filebeat: multiline: introduce merge by using max-lines as condition in stead of pattern **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 23, 2020, 10:04am UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434 "2020-04-23T10:04:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![williamd67](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@williamd67](https://discuss.elastic.co/u/williamd67)\ **Post date:** [April 23, 2020, 10:04am UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/1 "2020-04-23T10:04:09Z")
Once in a while people like to merge messages into a single line not based on a pattern but based on the number of lines that have to be merged. This may be caused by not having a clear usable pattern or by just wanting to reduce the number of lines in a message by combining several. There are situations that it may also be handy to combine the lines into a JSON-array that can be used by other applications. I propose to introduce an extra multiline parameter `kind` that distinguishes this behavior. Of course all the other parameters are still valid so in theory you can combine the `pattern` and the `max_lines` parameters. Although in practice I do not expect that. The values of the `kind` parameter would be `<>` (default and current implementation), `merge`, and `merge-json`, where `merge-json` will combine the messages in a JSON-array. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 23, 2020, 6:39pm UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/2 "2020-04-23T18:39:22Z")
Hi @williamd67, welcome to the Elastic community forums! This is an interesting suggestion. I'd like to make sure I understand the problem you're trying to solve first. Let's take the example from the [multiline documentation](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html): ```auto [beat-logstash-some-name-832-2015.11.28] IndexNotFoundException[no such index] at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver$WildcardExpressionResolver.resolve(IndexNameExpressionResolver.java:566) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:133) at org.elasticsearch.cluster.metadata.IndexNameExpressionResolver.concreteIndices(IndexNameExpressionResolver.java:77) at org.elasticsearch.action.admin.indices.delete.TransportDeleteIndexAction.checkBlock(TransportDeleteIndexAction.java:75) ``` My understanding of your use case is that you might want to only take the first, say, 3 lines of the above example and merge them into a single event. In that case So in this case you'd want a pattern to know where each multiline message starts and then the maximum # of lines you want to merge. But that could be accomplished with the `pattern`, `match`, and `max_lines` settings, right? Do we need to introduce the `kind: merge` option? That seems to be the current and default implementation anyway, right? ---
**Author:** ![williamd67](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@williamd67](https://discuss.elastic.co/u/williamd67)\ **Post date:** [April 24, 2020, 8:18am UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/3 "2020-04-24T08:18:56Z")
Hi @shaunak, Your case can indeed be done with the current multiline configuration. My case is a little different. It is when you know the number of lines of an event but there is no clear pattern. Per example someone has dumped a database table one field per line. In that case you know the number of lines for a row (= number of columns) but creating a pattern for that may be hard. In this situation the configuration can be as follows: ```auto multiline.kind: "merge" multiline.pattern: ".*" multiline.match: "before" multiline.negate: false multiline.max_lines: 13 ``` where 13 is the number of columns in a row. This will create a single event for a single row. In case you would choose `merge-json` they would be combined in one JSON-array. Another use-case is that someone just want to group a set of events that are similar. Per example the application is creating a lot of events and you want to put them in buckets of 300 each so that you can handle such group as a single event. In that case the configuration can be as follows: ```auto multiline.kind: "merge" multiline.pattern: ".*" multiline.match: "before" multiline.negate: false multiline.max_lines: 300 ``` A side-effect of the `merge` and `merge-json` options are that there are no lines discarded. I hope this clarifies the use-cases. ---
**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\ **Post date:** [April 24, 2020, 6:04pm UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/4 "2020-04-24T18:04:28Z")
Ah, thanks so much for clarifying, @williamd67. I think this is definitely worth considering as an enhancement! If you have a GitHub account, would you mind creating an enhancement issue for this at [https://github.com/elastic/beats/issues/new?template=feature-request.md](https://github.com/elastic/beats/issues/new?template=feature-request.md)? That way you can be notified of progress and be involved in any conversation with the team. If you prefer, I can do it on your behalf as well. Cheers, Shaunak ---
**Author:** ![williamd67](https://avatars.discourse-cdn.com/v4/letter/w/bc8723/32.png) [@williamd67](https://discuss.elastic.co/u/williamd67)\ **Post date:** [April 28, 2020, 8:07am UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/5 "2020-04-28T08:07:21Z")
Hi, I created an enhancement request for this [#18038](https://github.com/elastic/beats/issues/18038). Thanks for your support. BTW: I will be able to create a PR for this as well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2020, 8:14am UTC](https://discuss.elastic.co/t/filebeat-multiline-introduce-merge-by-using-max-lines-as-condition-in-stead-of-pattern/229434/6 "2020-05-26T08:14:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.