Dotnet core log - filter timestamp from log entry (filebeat, logstash)

Hi gang,

I'm importing logs from a dotnet core app using serilog and the output messages in the log file look like this. I need to work out out to extract the timestamp from the log entry line and use that as the @timestamp value to pass onto Logstash rather than using the default timestamp when the line was read.

The value I want to extra and use below is "2018-03-20 00:20:31.388 +02:00", do I need to use a grok filter in the file beats configuration which would extra the value and produce the new @timestamp field?

Thank you in advance!

{
"@version" => "1",
"host" => "8c94f1cff471",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"offset" => 25904,
"@timestamp" => 2018-03-29T11:57:29.167Z,
"fields" => {
"appenvironment" => "myapp",
"appsource" => "backend-service"
},
"message" => "2018-03-20 00:20:31.388 +02:00 [Information] Request finished in 12.7239ms 200 ",
"prospector" => {
"type" => "log"
},
"beat" => {
"version" => "6.2.3",
"name" => "8c94f1cff471",
"hostname" => "8c94f1cff471"
},
"source" => "/var/log/app/backend-kestrel-20180320.log"
}

Filebeat does not parse any contents. It's purpose is on shipping the logs.

Processing the timestamp can be done in Logstash. In Logstash you might also want to rename @timestamp to read_timestamp + add a receive_timestamp. This way you can compute latencies in your log-processing.

1 Like

is there a standard grok filter timestamp for this date format having the timezone as "+02:00" for example that I can use in logstash?

Why not parse the timezone into a separate field? Timezone information can be used to normalize timestamps to UTC-0, plus they can be useful for filtering.

The grok filter comes with 120+ predefined patterns, but I don't know if any of these predefined patterns captures the timezone as well.

Check out the grok filters documentation. It contains links to all predefined patterns + grokdebug/grokconstructor.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.