# Parsing multiline and capture data, also from 2nd line onwards

**URL:** <https://discuss.elastic.co/t/parsing-multiline-and-capture-data-also-from-2nd-line-onwards/51309>\
**Category:** Logstash\
**Created:** [May 30, 2016, 11:18am UTC](https://discuss.elastic.co/t/parsing-multiline-and-capture-data-also-from-2nd-line-onwards/51309 "2016-05-30T11:18:00Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![pandith\_asrar](https://avatars.discourse-cdn.com/v4/letter/p/ecccb3/32.png) [@pandith\_asrar](https://discuss.elastic.co/u/pandith_asrar)\
**Post date:** [May 30, 2016, 11:18am UTC](https://discuss.elastic.co/t/parsing-multiline-and-capture-data-also-from-2nd-line-onwards/51309/1 "2016-05-30T11:18:00Z")

</div>

Hi Magnus,

I am writing a config to read from a log file and using multiline codec.  
I am able to read the 1st line of the pattern and the multiline codec is also working, but the pattern for the multiline does not create any fields except for the first line.

Example:  
TRANSF\_2\_2\_1\> DBG\_21077 Create joiner cache on master relation : (Sun May 15 23:03:21 2016)  
TRANSF\_2\_2\_1\> TE\_7212 Increasing [Index Cache] size for transformation [**jnr\_PRODUCT\_FAMILY**] from [26157824] to [26188800].  
TRANSF\_2\_2\_1\> TE\_7212 Increasing [Data Cache] size for transformation [jnr\_PRODUCT\_FAMILY] from [52315648] to [52315896].

Pattern File Content:  
INFA\_TIMESTAMP %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}

LKP\_ST %{DATA:TRANS}:%{DATA:THREAD}\> %{WORD:MESSAGE\_CODE} SQL Query issued to database : (%{INFA\_TIMESTAMP:timestamp})  
LKP\_ED %{DATA:TRANS}:%{DATA:THREAD}\> %{WORD:MESSAGE\_CODE} Lookup cache creation completed : (%{INFA\_TIMESTAMP:timestamp})

JNR\_ST %{DATA:THREAD}\> %{DATA:MESSAGE\_CODE} Create joiner cache on master relation : (%{INFA\_TIMESTAMP:timestamp})  
JNR\_ED %{DATA:THREAD}\> %{DATA:MESSAGE\_CODE} Finished processing detail relation : (%{INFA\_TIMESTAMP:timestamp})

JNR\_ST\_NXT (?m)%{DATA:THREAD}\> %{DATA:MESSAGE\_CODE} Increasing [%{DATA}] size for transformation [%{WORD:TRANS}]  
JNR\_ED\_NXT (?m)%{DATA:THREAD}\> %{DATA:MESSAGE\_CODE} The index cache size that would hold [%{DATA}] input rows from the master for [%{WORD:TRANS}], in memory, is [%{DATA}] bytes

Config file content :

input {  
stdin {  
codec =\> multiline {  
patterns\_dir =\> "/var/lib/logstash/etc/grok"  
pattern =\> "(%{JNR\_ST\_NXT})|(%{JNR\_ED\_NXT})"  
negate =\> "false"  
what =\> "previous"  
}  
}  
}

filter {

grok {  
# match =\> { "message" =\> ["%{DATA:TRANS}:%{DATA:THREAD}\> %{WORD:Message\_code} SQL Query issued to database : (%{INFA\_TIMESTAMP:timestamp})"] }

```
                        match => { "message" => ["%{LKP_ST}","%{JNR_ST}"] }

                        patterns_dir => ["/var/lib/logstash/etc/grok"]
                        overwrite => ["message"]
                        add_tag => ["taskStarted"]
                        remove_tag => ["_grokparsefailure"]
                   }
    if "_grokparsefailure" in [tags] {

            grok {
                       # match => { "message" => ["%{DATA:TRANS}:%{DATA:THREAD}> %{WORD:Message_code} Lookup cache creation completed : \(%{INFA_TIMESTAMP:timestamp}\)"] }

                         match => { "message" => ["%{LKP_ED}","%{JNR_ED}"] }

                            patterns_dir => ["/var/lib/logstash/etc/grok"]
                            overwrite => ["message"]
                            add_tag => ["taskTerminated"]
                            remove_tag => ["_grokparsefailure"]
                    }
    } ##End of if statement

```

output {

stdout { codec =\> rubydebug }

}

Result:

message   
**TRANSF\_2\_2\_1\> DBG\_21077 Create joiner cache on master relation : (Sun May 15 23:03:21 2016)**  
TRANSF\_2\_2\_1\> TE\_7212 Increasing [Index Cache] size for transformation [jnr\_PRODUCT\_FAMILY] from [26157824] to [26188800].  
TRANSF\_2\_2\_1\> TE\_7212 Increasing [Data Cache] size for transformation [jnr\_PRODUCT\_FAMILY] from [52315648] to [52315896].

As can be see the message is built with all the three lines, but only the 1st line was used by the multiline codec to create the fields.  
Please let me know how can we get the fields from the pattern defined for the multiline codec to capture the fields from the 2nd line onwards.

Regards,  
Asrar

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:55am UTC](https://discuss.elastic.co/t/parsing-multiline-and-capture-data-also-from-2nd-line-onwards/51309/2 "2017-07-06T04:55:31Z")

</div>