How to replace logstash read time with log timing?

Elastic Stack Logstash
1

Hi, I am trying to configure logstash file which will replace the logstash read timestamp or system time with actual log time. In the kibana discover dashboard logs are displayed with logstash read time. How to replace this logstash read time. with actual log time.

  file {
    path => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/**/*.log"
    start_position => "beginning"
    type => "event"
  }
}

filter{
  date {
    match => [ "msg_timestamp", "yyyy-MM-dd'T'HH:mm:ss','SSS" ]
    target => "msgtime"
  }
      
  mutate {
    remove_field => [ "@timestamp" ]
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
  }
  stdout { codec => rubydebug }
}

actual log time - 2020-08-15T14:31:54,899 which is in the "message"

output

	{
      "@version" => "1",
       "message" => "2020-08-15T14:31:54,899 | ELAN-DpnInterface, ADD DPN 222845117715277 Instance e183ffcf-2a5b-4aa8-8821-33fbe508ea77",
          "path" => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/netvirt/netvirt.log",
       "@timestamp" => 2021-10-04T14:28:36.233Z,   --- logstash read time, laptop time 
          "type" => "event",
          "host" => "XXXXXXX"
   }
2

When you add a target it takes that value and puts it into that field. If you remove the target then it will put it in @timestamp. It sounds like you are looking to do this.

filter{
  date {
    match => [ "msg_timestamp", "yyyy-MM-dd'T'HH:mm:ss','SSS" ]
  }
}
3

You have not show any filter that creates the [msg_timestamp] field. If that field does not exist then the date filter is a no-op.

4

Thanks for the quick response. I tried what is suggested it did not work.

the output
' {
"message" => "2021-05-01T18:01:46,993 | ITM-TunnelInventoryState,REMOVE DTCN received for tun4fdba39c604",
"host" => "IN-00211777",
"type" => "event",
"@timestamp" => 2021-10-04T16:44:54.778Z,
"@version" => "1",
"path" => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/genius/genius.log"
}
'
My expectation of output is below

{
"message" => "2021-05-01T18:01:46,993 | ITM-TunnelInventoryState,REMOVE DTCN received for tun4fdba39c604", --- i want to remove the timestamp from the log messages
"host" => "IN-00211777",
"type" => "event",
"@timestamp" => 2021-05-01T18:01:46,993, ----- it must be the log timestamp
"@version" => "1",
"path" => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/genius/genius.log"
}

5

I was thinking you didn't post your entire conf but as @badger pointed out you aren't processing your message. You will need to parse your message field to break it down into individual fields before you use them. One method you can use is grok.

Something like this before you use the msg_timestamp field.

filter { 
    
 grok {
  match => { "message" => "%{DATA:msg_timestamp} \| %{GREEDYDATA:msg}" }
 }

 date {
  match => [ "msg_timestamp", "yyyy-MM-dd'T'HH:mm:ss','SSS" ]
 }
      
 mutate {
  remove_field => [ "message" ]
 }     

}

Output

{
    "msg": "ITM-TunnelInventoryState,REMOVE DTCN received for tun4fdba39c604",
    "msg_timestamp": "2021-05-01T18:01:46,993",
    "@timestamp": "   2021-05-01T22:01:46.993Z" <--- adjusted for timezone that's saved in Zulu time
}
6

Hi Badger,

thank you for the inputs. I used the 'timestamp' instead of 'msg_timestamp'. But still I see the below output. Log timestamp is not overriding in the output.

output:
{
"message" => "2021-05-03T20:19:22,084 | IFM-InterfaceInventoryState,REMOVE tun988ddd2ffe9",
"path" => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/genius/genius.log",
"host" => "IN-00211777",
"type" => "event",
"@version" => "1",
"@timestamp" => 2021-10-04T17:06:15.346Z
}

7

Use the filter I posted above. This should work.

filter { 
    
 grok {
  match => { "message" => "%{DATA:msg_timestamp} \| %{GREEDYDATA:msg}" }
 }

 date {
  match => [ "msg_timestamp", "yyyy-MM-dd'T'HH:mm:ss','SSS" ]
 }
      
 mutate {
  remove_field => [ "message" ]
 }     

}
8

Thank you for config. the filter file you provided is working fine. The only change required is the "timestamp" to be in the same time zone of msg_timestamp.

"host" => "IN-00211777",
"msg_timestamp" => "2021-05-03T20:19:26,610",
   "@timestamp" => 2021-05-03T14:49:26.610Z,
          "msg" => "Node added to oper ovsdb://uuid/8a932fd8-1e29-4d25-89c0-d97acd295df1/bridge/br-sdnc-sbi",
     "@version" => "1",
         "path" => "C:/Users/eaampnr/ELK/elk/csi/cic-1/log/events/ovsdb/ovsdb.log",
         "type" => "event"
9

All times in Elastic are stored in Zulu time and that's why you see the shift. Notice the Z at the end of the time.

If you need to set your timezone where the timezone in your message is generated from you can. But either way it will converted to Zulu time.

10

ok thank. we can close this topic.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.