# Multiline issue with 2 different patterns for a single event **URL:** \ **Category:** Logstash\ **Created:** [August 12, 2016, 8:16am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896 "2016-08-12T08:16:22Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 12, 2016, 8:16am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/1 "2016-08-12T08:16:23Z")
Hello I am trying to parse the log below through Logstash, however I can't seem to get the pattern to work. Essentially, I need to have the line straight after the dated line begining with _{0=Failed_ to be part of the same multiline event as the dated event above along with all the java output that begins with a space below it.. right up until the next dated log line. This is a sample of the log I am trying to parse through Logstash: ``` 2016-07-28 00:10:04,762 ERROR [qtp11076495-4340] com.pciwarehouse.gamediv.passimodutti.rest.passimoduttiRestImpl [writeDiagnosticMessage:1069] - Failed to post data to store FlameBoyance. Diagnostic data at 16CF/57/99/3F/4C/AC/1E/16CF-57993F4C-AC1E9A98-DCEA-AC1EA8EB-1F92-41BD0E. Failure for entries at position [0] {0=Failed to post data to store FlameBoyance - com.google.common.util.concurrent.UncheckedExecutionException: Portable(com.tangosol.util.WrapperException): (Wrapped: Failed request execution for FilterPartitionedPofCache service on Member(Id=4, Timestamp=2016-06-28 16:00:15.747, Address=172.30.168.235:10008, MachineId=28623, Location=site:KWSDev,machine:grdplo1001,process:2723,member:n4-grdplo1001.pciwarehouse.local, Role=pciwarehousegamedivpassimoduttiCacheServer)) null at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3) at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3) at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51) ... 1 more Caused by: Portable(java.lang.UnsupportedOperationException) at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57) at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3316) at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2604) at com.tangosol.io.pof.PortableException.readExternal(PortableException.java:150) at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:59) ... 12 more } 2016-07-28 00:10:04,490 INFO [qtp11076495-4340] com.pciwarehouse.gamediv.passimodutti.rest.resources.RestResource [time_functor:172] - http://hollana.htu.local:8082/chutneymarey/FindingStrings ``` And this is the filter I have in place: ``` filter { grok { match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{SYSLOG5424PRINTASCII:thread}%{SPACE}%{JAVACLASS:logger}%{SPACE}\[%{JAVAMETHOD:method}:%{NUMBER:line}]%{SPACE}-%{SPACE}%{GREEDYDATA:message}" } } date { match => ["timestamp", "MMM dd YYY HH:mm:ss", "MMM d YYY HH:mm:ss", "ISO8601"] remove_field => ["timestamp"] } multiline { pattern => "^\{" what => "previous" negate=> true } ``` } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 12, 2016, 10:43am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/2 "2016-08-12T10:43:22Z")
Try this multiline configuration: ```nohighlight multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } ``` Note that the multiline _filter_ is being deprecated. Prefer using the multiline codec. Also, to overwrite the existing `message` field in your grok filter you need to set `overwrite => ["message"]`. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 12, 2016, 11:28am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/3 "2016-08-12T11:28:06Z")
It's looking much better than it did before, though I'm still getting a \__grokparsefailure_ ``` "message" => "2016-07-28 00:10:04,762 ERROR [qtp11076495-4340] com.pciwarehouse.gamediv.passimodutti.rest.passimoduttiRestImpl [writeDiagnosticMessage:1069] - Failed to post data to store FlameBoyance. Diagnostic data at 16CF/57/99/3F/4C/AC/1E/16CF-57993F4C-AC1E9A98-DCEA-AC1EA8EB-1F92-41BD0E. Failure for entries at position [0]\nFailed to post data to store FlameBoyance. Diagnostic data at 16CF/57/99/3F/4C/AC/1E/16CF-57993F4C-AC1E9A98-DCEA-AC1EA8EB-1F92-41BD0E. Failure for entries at position [0]\n{0=Failed to post data to store FlameBoyance - com.google.common.util.concurrent.UncheckedExecutionException: Portable(com.tangosol.util.WrapperException): (Wrapped: Failed request execution for FilterPartitionedPofCache service on Member(Id=4, Timestamp=2016-06-28 16:00:15.747, Address=172.30.168.235:10008, MachineId=28623, Location=site:KWSDev,machine:grdplo1001,process:2723,member:n4-grdplo1001.pciwarehouse.local, Role=pciwarehousegamedivpassimoduttiCacheServer)) null\n at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.PartitionedService.onNotify(PartitionedService.CDB:3)\n at com.tangosol.coherence.component.util.daemon.queueProcessor.service.grid.partitionedService.PartitionedCache.onNotify(PartitionedCache.CDB:3)\n at com.tangosol.coherence.component.util.Daemon.run(Daemon.CDB:51)\n ... 1 more\nCaused by: Portable(java.lang.UnsupportedOperationException)\n at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:57)\n at com.tangosol.io.pof.PofBufferReader.readAsObject(PofBufferReader.java:3316)\n at com.tangosol.io.pof.PofBufferReader.readObject(PofBufferReader.java:2604)\n at com.tangosol.io.pof.PortableException.readExternal(PortableException.java:150)\n at com.tangosol.io.pof.ThrowablePofSerializer.deserialize(ThrowablePofSerializer.java:59)\n ... 12 more\n}", "@version" => "1", "@timestamp" => "2016-07-27T23:10:04.762Z", "host" => "grdvls1004.pciwarehouse.local", "loglevel" => "ERROR", "thread" => "[qtp11076495-4340]", "logger" => "com.pciwarehouse.gamediv.passimodutti.rest.passimoduttiRestImpl", "method" => "writeDiagnosticMessage", "line" => "1069", "tags" => [ [0] "_grokparsefailure", [1] "multiline" ] } { "message" => "2016-07-28 00:10:04,490 INFO [qtp11076495-4340] com.pciwarehouse.gamediv.passimodutti.rest.resources.RestResource [time_functor:172] - http://hollana.htu.local:8082/chutneymarey/FindingStrings\nhttp://hollana.htu.local:8082/chutneymarey/FindingStrings", "@version" => "1", "@timestamp" => "2016-07-27T23:10:04.490Z", "host" => "grdvls1004.pciwarehouse.local", "loglevel" => "INFO", "thread" => "[qtp11076495-4340]", "logger" => "com.pciwarehouse.gamediv.passimodutti.rest.resources.RestResource", "method" => "time_functor", "line" => "172" } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 12, 2016, 11:30am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/4 "2016-08-12T11:30:07Z")
I suspect you have an extra grok filter somewhere, because the grok filter you're showing above is obviously successful. Are there any unexpected files (e.g. backup files) in /etc/logstash/conf.d or wherever you store your configuration files? ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 12, 2016, 11:42am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/5 "2016-08-12T11:42:15Z")
At the moment, I'm just running it manually via STDIN and observing the output on STDOUT while I test that the configuration works OK. This is the full test configuration I am using to test that the parsing: ``` input { stdin { } } filter { grok { match => { "message" => "\A%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{SYSLOG5424PRINTASCII:thread}%{SPACE}%{JAVACLASS:logger}%{SPACE}\[%{JAVAMETHOD:method}:%{NUMBER:line}]%{SPACE}-%{SPACE}%{GREEDYDATA:message}" } overwrite => ["message"] } date { match => ["timestamp", "MMM dd YYY HH:mm:ss", "MMM d YYY HH:mm:ss", "ISO8601"] remove_field => ["timestamp"] } multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" } } output { stdout { codec => rubydebug } } ``` And then manually running it with: /bin/logstash -f ../testconfigs/multiline.conf ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 12, 2016, 11:57am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/6 "2016-08-12T11:57:28Z")
Put the multiline filter first (and switch to the multiline codec). ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 12, 2016, 1:44pm UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/7 "2016-08-12T13:44:27Z")
Putting the multiline filter at the beginning did the trick !! Thank you !! > [@magnusbaeck](#): > > (and switch to the multiline codec) Thanks - I'll now switch to the multiline codec. Regarding the multiline codec - does it have to be used under the input section of a configuration of a file or can it also be used under the filter section as well? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 12, 2016, 2:00pm UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/8 "2016-08-12T14:00:29Z")
Codecs are for inputs and/or outputs only. They can't be used as filters. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 16, 2016, 10:40am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/9 "2016-08-16T10:40:23Z")
Hello I am currently using filebeat to input the logs onto logstash. My current input configuration is as shown below. ``` input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" } } ``` I receive various application log files of different formats from multiple servers). **What is the best way to integrate the multiline codec into the input?** Would something like be suitable? All the log files do generally start with an ISO Timestamp. ``` input { beats { port => 5044 ssl => true ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601} " negate => true what => previous } } } ``` * * * Just an FYI, I address the type of files on each filebeat configuration (filebeat.yml) as follows: ``` - paths: - /app/vertuso/log/logs/VERTUSOCache_VERTUSOCacheServer*backend.log fields: applog: VERTUSOCacheBackend environment: dev fields_under_root: true document_type: VERTUSOCacheBackend close_older : 1m - paths: - /app/vertuso/log/logs/VERTUSOCache_Main* fields: applog: VERTUSOCacheProxy environment: dev fields_under_root: true document_type: VERTUSOCacheProxy close_older : 1m - paths: - /app/vertuso/log/coherence-server-n* fields: applog: coherence-server-instance environment: dev fields_under_root: true document_type: coherence-server-instance close_older: 1m - paths: - /app/vertuso/log/coherence-server-proxy* fields: applog: coherence-server-proxy environment: dev fields_under_root: true document_type: coherence-server-proxy close_older: 1m - paths: - /app/vertuso/log/coherence-server-mbean* fields: applog: coherence-server-mbean environment: dev fields_under_root: true document_type: coherence-server-mbean close_older: 1m ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2016, 10:46am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/10 "2016-08-16T10:46:03Z")
It is often beneficial to perform the multiline processing as close to the source as possible, and FileBeat now supports multiline processing. Have you considered moving this logic from Logstash to FileBeat? ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 16, 2016, 10:57am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/11 "2016-08-16T10:57:03Z")
Ah sure, I am happy to move it to filebeat. Would something like this work? Essentially, telling it what kind of log files need multipath - And not forcing multipath on all the files being sent by logstash. ``` - paths: - /app/vertuso/log/logs/VERTUSOCache_VERTUSOCacheServer*backend.log fields: applog: VERTUSOCacheBackend environment: dev fields_under_root: true document_type: VERTUSOCacheBackend close_older : 1m multiline: pattern: ^%{TIMESTAMP_ISO8601} negate: true match: after - paths: - /app/vertuso/log/logs/VERTUSOCache_Main* fields: applog: VERTUSOCacheProxy environment: dev fields_under_root: true document_type: VERTUSOCacheProxy close_older : 1m multiline: pattern: ^%{TIMESTAMP_ISO8601} negate: true match: after - paths: - /app/vertuso/log/coherence-server-n* fields: applog: coherence-server-instance environment: dev fields_under_root: true document_type: coherence-server-instance close_older: 1m - paths: - /app/vertuso/log/coherence-server-proxy* fields: applog: coherence-server-proxy environment: dev fields_under_root: true document_type: coherence-server-proxy close_older: 1m - paths: - /app/vertuso/log/coherence-server-mbean* fields: applog: coherence-server-mbean environment: dev fields_under_root: true document_type: coherence-server-mbean close_older: 1m ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 16, 2016, 11:13am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/12 "2016-08-16T11:13:05Z")
AFAIK Filebeat's multiline exprsesions don't support Logstash's grok patterns but otherwise it looks okay. I believe the documentation contains details about the regexp flavor that's supported. ---
**Author:** ![jinal.shah](https://avatars.discourse-cdn.com/v4/letter/j/3da27b/32.png) [@jinal.shah](https://discuss.elastic.co/u/jinal.shah)\ **Post date:** [August 17, 2016, 1:13pm UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/13 "2016-08-17T13:13:16Z")
Hello Just, wanted to thank you for all your assistance, suggestions and corrections with this query! It's all working fantastically well. I am now using filebeat to parse the multiline at the source itself after ammending the timestamp using regexp as follows: ``` multiline: pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}' negate: true match: after ``` Once again, your help has been very much appreciated! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:43am UTC](https://discuss.elastic.co/t/multiline-issue-with-2-different-patterns-for-a-single-event/57896/14 "2017-07-06T04:43:02Z")