I'm trying to use Logstash to replace a custom, overly complex system that I've inherited.
It currently outputs the timestamp of the event into a field called "@ts", but this is stored in epoch_millis format. I'm trying to get Logstash to output the timestamp in this date format.
Here is the relevant field from the mapping:
"@ts": {"format": "epoch_millis", "type": "date"}
And the raw JSON documents when I retrieve them look like this:
"fields": {"@ts": [1508832196000]}
No matter what I do, I cannot seem to get Logstash to output the date format in epoch_millis format, and all the documents are discarded when written to that index. If I try to write to a different index with a mapping of strict_date_optional_time then all works perfectly, but the documents are stored in the standard format e.g. "2017-10-24T08:26:09.000Z"
Is there any way to get logstash to output a date in epoch_millis format for compatibility?
I've tried https://github.com/wiibaa/logstash-filter-date_formatter but Joda Time doesn't seem to support Unix timestamps.
Any pointers here? Thanks!
Edit:
I've tried using Ruby to set @timestamp to this, but that results in an error:
event.set('@timestamp',event.get('@ts').to_i)
Ruby exception occurred: wrong argument type Fixnum (expected LogStash::Timestamp)
But it looks like I can set @ts to to this
event.set('@ts',event.get('@ts').to_i)
Guessing @timestamp here is a reserved name and can't be overridden.
Still doesn't solve the problem though as something is casting the integer timestamp to a date somewhere.