# How do I create a filter for a custom app log with different lines? **URL:** \ **Category:** Logstash\ **Created:** [July 20, 2016, 10:04am UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961 "2016-07-20T10:04:51Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 10:04am UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/1 "2016-07-20T10:04:51Z")
Hello I am knew to Logstash/Elasticsearch.. I have a log file with multiple lines within it. What's the best way to create a filter for a log similar to the example below? 2016-07-12T02:04:32.998+0100: Total time for which application threads were stopped: 0.0179270 seconds 2016-07-12T02:04:33.471+0100: [CMS-concurrent-sweep: 0.473/0.473 secs] [Times: user=0.48 sys=0.00, real=0.47 secs] 2016-07-12T02:04:33.494+0100: [CMS-concurrent-reset: 0.022/0.022 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 2016-07-12T02:04:35.494+0100: Application time: 2.4959740 seconds 2016-07-12T02:04:35.496+0100: [GC [1 CMS-initial-mark: 1143120K(1572864K)] 1144381K(2044736K), 0.0037850 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] Thanks ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 10:12am UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/2 "2016-07-20T10:12:53Z")
This is currently my conf file: input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key" } } filter { if [source] =~ "filename1.log" { grok { match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{DATA:datacode}%{SPACE}%{JAVACLASS:javaclass}%{SPACE}%{DATA:dataoperation}%{SPACE}%{NOTSPACE}%{SPACE}%{CISCO\_REASON:code}%{DATA}%{SPACE}%{GREEDYDATA:lumpdata}" } } date { match =\> ["time" , "dd/MMM/yyyy:HH:mm:ss Z"] } } else if [source] =~ "filename2.log" { grok { commentedline.. match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601:time}\W\s%{DATA:data}\W\s%{GREEDYDATA:seconds}" } match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601}%{NOTSPACE}%{SPACE}%{SYSLOG5424SD}%{SPACE}%{GREEDYDATA}" } } } } output { stdout { } elasticsearch { hosts =\> ["localhost:9200"] sniffing =\> true manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 20, 2016, 11:41am UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/3 "2016-07-20T11:41:28Z")
I'd use the aggregate filter. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 12:29pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/4 "2016-07-20T12:29:10Z")
Hi Magnus Many thanks - that plugin looks quite interesting. Though in my case, the log lines aren't linked to the same task - would it still work? Also, how would you define the grok for different lines as the log lines are very different to each other? Thanks in advance! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 20, 2016, 12:35pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/5 "2016-07-20T12:35:37Z")
> Though in my case, the log lines aren't linked to the same task - would it still work? Not sure what you mean here. Don't you want to emit a single events with all numbers? Or was your question just about how to parse these different lines and emit multiple events? > Also, how would you define the grok for different lines as the log lines are very different to each other? A single grok filter can list multiple expressions that'll get tried in order. There's an example of this in the documentation. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 1:01pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/6 "2016-07-20T13:01:13Z")
Hi Magnus > Or was your question just about how to parse these different lines and emit multiple events? That is correct, I was looking to parse these different lines and emit multiple events. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 20, 2016, 1:17pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/7 "2016-07-20T13:17:34Z")
Okay, never mind the aggregate filter then. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 1:58pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/8 "2016-07-20T13:58:52Z")
> [@magnusbaeck](#): > > A single grok filter can list multiple expressions that'll get tried in order. There's an example of this in the documentation. For using multiple expressions, do you add multiple match lines under grok or do all the expressions go within one match line? So for example, to match a single line: grok { match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{DATA:datacode}%{SPACE}%{JAVACLASS:javaclass}%{SPACE}%{DATA:dataoperation}%{SPACE}%{NOTSPACE}%{SPACE}%{CISCO\_REASON:code}%{DATA}%{SPACE}%{GREEDYDATA:lumpdata}" } } OR To Match 2 different lines of a log file: grok { match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{DATA:datacode}%{SPACE}%{JAVACLASS:javaclass}%{SPACE}%{DATA:dataoperation}%{SPACE}%{NOTSPACE}%{SPACE}%{CISCO\_REASON:code}%{DATA}%{SPACE}%{GREEDYDATA:lumpdata}" } match =\> { "message" =\> "\A%{TIMESTAMP\_ISO8601:time}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{GREEDYDATA:datacode} } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 20, 2016, 2:36pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/9 "2016-07-20T14:36:50Z")
See the example in [the documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match) prefixed by "If you need to match multiple patterns against a single field, the value can be an array of patterns". ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [July 20, 2016, 2:53pm UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/10 "2016-07-20T14:53:54Z")
Perfect, thank you very much! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:47am UTC](https://discuss.elastic.co/t/how-do-i-create-a-filter-for-a-custom-app-log-with-different-lines/55961/11 "2017-07-06T04:47:15Z")