I am using multiline codec in my logstash config file in input section to merge few lines of log file. Lines 5,6,7,8,9 are getting merged into a single line and the same is visible in Kibana screen for message and sub_class fields(refer config file).
My requirement is to read the week consumption and month consumption and show it in new scripted field Consumption Data. I am not able to get any value for the merged line using scripted field.
I used below script and received ‘Value not present, for the merged line and ‘Value Present’ for all other lines.
Script -
if(doc['sub_class.keyword'].size()==0) return 'Value not present'; else return 'Value Present';
Log File -
- 2020/05/12_09:25:04,Attr 0x0000
- 2020/05/12_09:25:04, Attr 0x0000
- 2020/05/12_09:25:04, Attr 0x0000
- 2020/05/12_09:25:04, Attr 0x0000
- 2020/05/12_09:25:04, Daily Consumption report received
- 2020/05/12_09:25:04,Current week consumption was 100 (avail=1)
- 2020/05/12_09:25:04,Day is not Monday, so add current day into current week
- 2020/05/12_09:25:04,Current month consumption was 5000 (avail=1)
- 2020/05/12_09:25:04,Day is not Monday, so add current day into current week
- 2020/05/12_09:25:04, Attr 0x0000
- 2020/05/12_09:25:04, Attr 0x0000
- 2020/05/12_09:25:04, Attr 0x0000
Logstash Config file -
input {
file {
path => ["---------------------------------/WHOHealth.txt"]
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline {
pattern => "(^.+Current week .+)|(^.+Day is .+)|(^.+Current month.+)”
what => "previous"
negate => "false"
}
}
}
filter {
if ([message] =~ "{"index") {
drop {}
} else if ([message] =~ "month") {
grok {
match => {"message" => "%{GREEDYDATA:log_line}"}
}
mutate {
add_field => { "sub_class" => "%{[log_line]}" }
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "TestIndex"
}
}