How to retain the time zone and nanoseconds in timestamp when migrating es through logstash

Elastic Stack Logstash
1

In the old cluster, the timestamp is "@timestamp": "2024-08-12T17:40:31.098422413+08:00". For stability reasons, I still want to keep it after migrating to the new cluster. I have tried many filters like behind, but none of them work. The new cluster has always been "@timestamp": "2024-08-12T09:40:31.098Z". I don't know if it's because there is no match or some other reason.How to retain the time zone and nanoseconds format

date {
    match => [ "@timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ" ]
    timezone => "Asia/Shanghai"
    target => "@timestamp"
  }
2

This was explained here.

Credits to Badger

Here is a sample:

input {
 generator { 
        # message =>   [ "2024-08-12T09:40:31.098Z" ]
        message =>   [ "2024-08-12T17:40:31.098422413+08:00" ]
        count => 1
 }
}
output {
 stdout {codec => rubydebug}
}
filter {

date {
    match => [ "message",  "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ"] # "ISO8601"
    timezone => "Asia/Shanghai"
    target => "@timestamp"
  }
  ruby {
    code => "event.set('[longformat]', LogStash::Timestamp.new(event.get('[message]')) )"
  }
     mutate {  remove_field => ["@version", "event", "host"] }
}

Output:

{
       "message" => "2024-08-12T17:40:31.098422413+08:00",
    "@timestamp" => 2024-08-12T09:40:31.098Z,
    "longformat" => 2024-08-12T09:40:31.098422413Z
}
3

Thank you for your suggestion to keep the nano format. I still want to know how to keep the time zone. I added the Shanghai time zone and it didn't work. I want to keep +8:00. The purpose is to keep the format of the full data and incremental data consistent to avoid possible problems in the future. Thank you very much for your advice.

4

If you need date and time before 08:00:

filter {

dissect { mapping => {
        "message" => "%{tmpmsg}+%{}"
      }
}
 mutate {
        add_field => { "timeshanghai" => "%{[tmpmsg]}+00:00Z" }
    }

		
date {
    match => [ "message",  "yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSSZ"] # "ISO8601"
    timezone => "Asia/Shanghai"
    target => "@timestamp1"
  }
  
  ruby {
    code => "event.set('[@timestamp]', LogStash::Timestamp.new(event.get('[timeshanghai]')) )"
  } 
     mutate {  remove_field => ["@version", "event", "host", "timeshanghai", "tmpmsg" ] }
	 
}

Output:

{
     "@timestamp" => 2024-08-12T17:40:31.098422413Z,
        "message" => "2024-08-12T17:40:31.098422413+08:00",
    "@timestamp1" => 2024-08-12T09:40:31.098Z
}