I have been using the generated @timestamp field in our documents as a record of when a document was sent to Elastic. I have been renaming the field in Logstash to suit our document structure.
Our version of Elastic is such that the precision of the timestamp is to milliseconds, and this matches the other timestamps in the document.
Upgrading to version 8, I see that the @timestamp precision is now to the microseconds level, due to a change from a Ruby library to a Java library I think.
I want to keep the precision of this timestamp at milliseconds. What's the simplest way to do this? I could do it in 2 steps like this:
Thanks for that. That's true, though I am already removing other fields from the message so the mutate filter will be there anyway. I would probably prefer to group all the removes together instead of splitting over 2 different filters.
I guess I was trying to see if there was a way of changing the precision of the timestamp without using a dedicated Ruby filter.
Thanks again. The field is being mapped to the "date" datatype, which I can see is in milliseconds precision already. So queries that can be satisfied by returning the indexed data would be OK, and it's those requests that are looking at the returned source document that would run into the microseconds.
So the timestamp2 is now at milliseconds, which is good, but also has changed timezone but I want to keep it in UTC. It's probably very simple, but how can I keep timestamp2 in UTC?
t = @timestamp = current time in UTC
t.utc_offset = 11 hours
(t - t.utc_offset) = current time in UTC - 11 hours (some complete other timezone)
(t - t.utc_offset).strftime = current time in UTC, formatted correctly
So it seems to work by backshifting the time by the tz difference, to negate the later tz shift forward. It seems odd that strftime would do that time shifting?
I also tried doing it using mutate to create new fields without using timestamp, and it sort of worked, but to get the "Z" tz indicator I had to hard code it, although I guess there's another way to do that. But I just realised the date format I used in the Ruby filter also hard codes the Z I think...
Yeah, sort of. But in the elastic stack we live in a world where everything has to be UTC. That's not where most folks live. I can definitely see why some people would think time-shifting in the library is useful.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.