We have log file in which we have to capture the first line matching "TIMESTAMP_ISO8601" against build_StartTime filed and last line matching "TIMESTAMP_ISO8601" against build_EndTime filed. After this we have to calculate the difference and should store the difference against duration filed. We tried with date filter but that's not working out, Please help us
Log File:
[2019-01-31 21:28:22Z INFO Program] Version: 2.122.1
some logs in the middle
[2019-01-31 21:28:45Z INFO Worker] Job completed.
Below is our current Logstash config File:
input {
beats {
client_inactivity_timeout => 1200
port => 5002
}
}
filter
{
grok {
add_tag => [ "start" ]
match => { "message" => "%{TIMESTAMP_ISO8601:build_StartTime}" }
}
date {
match => ["build_StartTime", "ISO8601"]
target => "build_StartTime"
}
mutate {
remove_field => [ "message" ]
}
}
output {
elasticsearch {
hosts => ["elasticsearch:9200"]
index => "tfslog-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
LOGSTASH OUTPUT:
"_index": "tfslog-2019.02.21",
"_type": "doc",
"_id": "-dKUD2kBAWdIEry4JFmb",
"_version": 1,
"_score": null,
"_source": {
"tags": [
"beats_input_codec_plain_applied",
"start",
"_dateparsefailure"
],
"beat": {
"hostname": "RMD-VRA103029",
"version": "6.3.0",
"name": "RMD-VRA103029"
},
"@version": "1",
"offset": 19709,
"@timestamp": "2019-02-21T10:21:39.153Z",
"source": "C:\TFS2017_Agent\_diag\Worker_20190134-00015-utc.log",
"host": {
"name": "RMD-VRA103029"
},
"build_StartTime": "2019-01-31 21:28:22Z"
},
"fields": {
"@timestamp": [
"2019-02-21T10:21:39.153Z"
]
},
"sort": [
1550744499153
]