Change timestamp in logstash

Elastic Stack Logstash
1

Im having logs like below from logstash 8.9.1

2024-03-06T23:43:37.317829530Z 192.168.1.132 <14> 2024-03-06T23:43:37Z forwarder.tmes.trendmicro.com tmes[1]: CEF:0|Trend Micro|TMES|1.0.0.0|100101|DETECTION|6|rt=2024-03-06T23:43:12.149Z

how to change the first timestamp to other local timezone?

2

Is it possible

3

Yes it's possible.
Parse with grok/dissect then use the date plugin.

date {
    match => ["timestamp", "ISO8601"]
	timezone => "Asia/Dubai" # set your own accoding to Joda time https://joda-time.sourceforge.net/timezones.html
	target=> "@timestamp" # or any other field
}
4

The first timestamp is the field produced after the logs saved in the file so will the grok works on it?

5

The first timestamp is the field produced after the logs saved in the file so will the grok works on it?

For example my log is hello

output will be like

{Timestamp} {host ip} {hello}

So how the grok look upon it

6

%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:ip} %{GREEDYDATA:msg}
That will produce:

timestamp 2024-03-06T23:43:37.317829530Z
ip 192.168.1.132
msg <14>·2024-03-06T23:43:37Z·forwarder.tmes.trendmicro.com·tmes[1]:·CEF:0
8

SO the configuration only works if there is time stamp available in log is there any other way to add the timestamp +5:30 hours for it

9

If not receive time as @timestamp as a field, LS will take the host local time for @timestamp.
Since you are receiving as a part of log messages, you will parse and with the date plugin you will overwrite @timestamp with the log time. Like above.

If you not receive as the part of message or you don't want to use then set @timestamp to another time zone or hardcoded TZ- actually offset, like you suggested +5:30. Be aware, the hard coded value doesn't take in account daylight saving time.

if ![timestamp] { # if timestamp field does not exist
  mutate {  # maybe you will need to convert string
        convert => {  "@timestamp" => "string"}
  }
  date {
    match => ["@timestamp", "ISO8601"]
    timezone => "Asia/Dubai"  
  }

}
10

The +5:30 is ok for us beacause those logs moved for just test monitoring backup purpose which plugin will be used for this ?

12

Date offset +5:30 will be used for all events/messages if the timestamp field doesn't exist. If you more granular add one or a few more IFs.

13

This is my log now pushing to another server

2024-03-08T08:51:37.382895768Z 2.10.14.14 <14> 2024-03-08T08:51:37Z forwarder.tmes.trendmicro.com tmes[1]: CEF:0|Trend Micro|TMES|1.0.0.0|100101|DETECTION|6|rt=2024-03-08T08:50:58.248Z cs1Label=eventType cs1=scan_limitation cs2Label=domainName cs2=ll.com suser=rll.com duser=llk.com cs3Label=direction cs3=incoming cs4Label=messageId cs4=<CACtuar2zHwS9mf\=y5JJZw-wH4K-ll.com> msg=Adangal - 07-Mar-2024, 16:49 cn1Label=messageSize cn1=908230 cs5Label=policyName cs5=CFE2022:IVP: Virus_Policy act=Bypass cs6Label=details cs6={}`

I Want the first timestamp to add +5:30 hours thats my case

my grok is

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp1} %{DATA} %{TIMESTAMP_ISO8601:timestamp2} %{GREEDYDATA:program}" }
}

I tried with date and ruby not works for me

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.