Sorting string value stored in field 'message'

Elastic Stack Elasticsearch
1

Hello,

I'm currently parsing our logs using using Logstash's multiline filter to combine entries with the same reference e,g,
P1-hrrnjee8

Message Field Value:
example 1
2016-02-15 00:00:05 P1-hrrnjee8:None processMethod None
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 New session 1-1939c534
2016-02-15 00:00:08 P1-hrrnjee8:P1-hrrnjee8 Finished 7292 bytes (0.1821s)
2016-02-15 00:00:14 P1-hrrnjee8:None InHeads: {'UPGRADE_INSECURE_REQUESTS': '1'
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 Web handlePage:
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 Msgs: [] [] [] {}

As you can see, this is working just fine. However, I need the message value to also be displayed with the timestamp in ascending order e.g.

example 2
2016-02-15 00:00:05 P1-hrrnjee8:None processMethod None
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 New session 1-1939c534
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 Web handlePage:
2016-02-15 00:00:05 P1-hrrnjee8:P1-hrrnjee8 Msgs: [] [] [] {}
2016-02-15 00:00:08 P1-hrrnjee8:P1-hrrnjee8 Finished 7292 bytes (0.1821s)
2016-02-15 00:00:14 P1-hrrnjee8:None InHeads: {'UPGRADE_INSECURE_REQUESTS': '1'

Is there anyway I can achieve example 2 when sending the parsed data to elasticsearch.

In summary, I just need it to look like example 2 when being viewed in kibana. It doesn't necessarily have to be stored that way.

2

I ended up using the ruby filter in conjunction with the multiline. Multiline groups all our logs based on the reference found on each log line.

multiline {
stream_identity => "%{requestreference}"
pattern => "."
what => "previous"
}

Initially, each line is split and stored into an array. We append a new line delimiter to each index position. We then sort on the time and then join the lines back together to form the message in the correct sequence
Please note: 10.chr was used because escaping for a newline i.e. \n was adding the literal... \n in the message field

ruby {code => "event['message'] = event['message'].split(/\r?\n/)
event['message'].map! {|logline| logline + 10.chr}
event['message'] = event['message'].sort_by {|el| el.scan(/\d+-\d+-\d+\s\d+:\d+:\d+/)}
event['message'] = event['message'].join"
}

I hope this helps someone :smile_cat: