How to handle multi line logs in logstash


(Parvatayya Malimath) #1

Hi

I have such logs, i am not sure how handle this multi line logs, as you can see. its not consistent, we have multi lines at times and single lines at times.
please help, how to parse them

2018-10-22 15:32:24.407 DEBUG [task-detail-service-dev,9d3c34d063a7c5fe,646dff3d87b59c21,true] 13287 --- [askExecutor-438] CCMessagePublisher

2018-10-22 15:32:24.407 DEBUG [task-detail-service-dev,9d3c34d063a7c5fe,646dff3d87b59c21,true] 13287 --- [askExecutor-438] CCMessagePublisher : Publishing message some message to seome_event_centre:
taskInstanceId: "111c0242-d5e7-11e8-b966-0050568b4ef6"
eventType: "TaskVariablesUpdated"
variables {
key: "SOME_KEY_1"
value {
stringValue: "SOME_VALUE_1"
}
}
variables {
key: "SOME_KEY_2"
value {
stringValue: "SOME_VALUE_2"
}
}

I am using below conf

input { stdin { } }

filter {
grok {
patterns_dir => ["./patterns"]
match => { "message" => "%{WORD:log_level} [%{GREEDYDATA:local_address},%{POSTFIX_QUEUEID:queue_id},%{POSTFIX_QUEUEID:spanId},%{WORD:random_entry}]" }
}

date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}

output {
stdout { codec => rubydebug }
}


#2

use the multiline codec when importing, normally you would use the timestamp as the pattern to start a new event.


(Magnus B├Ąck) #3

See https://www.elastic.co/guide/en/logstash/current/multiline.html.


(Parvatayya Malimath) #4

thanks it worked


(Parvatayya Malimath) #5

thanks a lot, it works


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.