Hi Magnus,
I am writing a config to read from a log file and using multiline codec.
I am able to read the 1st line of the pattern and the multiline codec is also working, but the pattern for the multiline does not create any fields except for the first line.
Example:
TRANSF_2_2_1> DBG_21077 Create joiner cache on master relation : (Sun May 15 23:03:21 2016)
TRANSF_2_2_1> TE_7212 Increasing [Index Cache] size for transformation [jnr_PRODUCT_FAMILY] from [26157824] to [26188800].
TRANSF_2_2_1> TE_7212 Increasing [Data Cache] size for transformation [jnr_PRODUCT_FAMILY] from [52315648] to [52315896].
Pattern File Content:
INFA_TIMESTAMP %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
LKP_ST %{DATA:TRANS}:%{DATA:THREAD}> %{WORD:MESSAGE_CODE} SQL Query issued to database : (%{INFA_TIMESTAMP:timestamp})
LKP_ED %{DATA:TRANS}:%{DATA:THREAD}> %{WORD:MESSAGE_CODE} Lookup cache creation completed : (%{INFA_TIMESTAMP:timestamp})
JNR_ST %{DATA:THREAD}> %{DATA:MESSAGE_CODE} Create joiner cache on master relation : (%{INFA_TIMESTAMP:timestamp})
JNR_ED %{DATA:THREAD}> %{DATA:MESSAGE_CODE} Finished processing detail relation : (%{INFA_TIMESTAMP:timestamp})
JNR_ST_NXT (?m)%{DATA:THREAD}> %{DATA:MESSAGE_CODE} Increasing [%{DATA}] size for transformation [%{WORD:TRANS}]
JNR_ED_NXT (?m)%{DATA:THREAD}> %{DATA:MESSAGE_CODE} The index cache size that would hold [%{DATA}] input rows from the master for [%{WORD:TRANS}], in memory, is [%{DATA}] bytes
Config file content :
input {
stdin {
codec => multiline {
patterns_dir => "/var/lib/logstash/etc/grok"
pattern => "(%{JNR_ST_NXT})|(%{JNR_ED_NXT})"
negate => "false"
what => "previous"
}
}
}
filter {
grok {
# match => { "message" => ["%{DATA:TRANS}:%{DATA:THREAD}> %{WORD:Message_code} SQL Query issued to database : (%{INFA_TIMESTAMP:timestamp})"] }
match => { "message" => ["%{LKP_ST}","%{JNR_ST}"] }
patterns_dir => ["/var/lib/logstash/etc/grok"]
overwrite => [ "message" ]
add_tag => [ "taskStarted" ]
remove_tag => ["_grokparsefailure"]
}
if "_grokparsefailure" in [tags] {
grok {
# match => { "message" => ["%{DATA:TRANS}:%{DATA:THREAD}> %{WORD:Message_code} Lookup cache creation completed : \(%{INFA_TIMESTAMP:timestamp}\)"] }
match => { "message" => ["%{LKP_ED}","%{JNR_ED}"] }
patterns_dir => ["/var/lib/logstash/etc/grok"]
overwrite => [ "message" ]
add_tag => [ "taskTerminated"]
remove_tag => ["_grokparsefailure"]
}
} ##End of if statement
output {
stdout { codec => rubydebug }
}
Result:
message
TRANSF_2_2_1> DBG_21077 Create joiner cache on master relation : (Sun May 15 23:03:21 2016)
TRANSF_2_2_1> TE_7212 Increasing [Index Cache] size for transformation [jnr_PRODUCT_FAMILY] from [26157824] to [26188800].
TRANSF_2_2_1> TE_7212 Increasing [Data Cache] size for transformation [jnr_PRODUCT_FAMILY] from [52315648] to [52315896].
As can be see the message is built with all the three lines, but only the 1st line was used by the multiline codec to create the fields.
Please let me know how can we get the fields from the pattern defined for the multiline codec to capture the fields from the 2nd line onwards.
Regards,
Asrar