Output date in epoch_millis format

I'm trying to use Logstash to replace a custom, overly complex system that I've inherited.

It currently outputs the timestamp of the event into a field called "@ts", but this is stored in epoch_millis format. I'm trying to get Logstash to output the timestamp in this date format.

Here is the relevant field from the mapping:
"@ts": {"format": "epoch_millis", "type": "date"}

And the raw JSON documents when I retrieve them look like this:
"fields": {"@ts": [1508832196000]}

No matter what I do, I cannot seem to get Logstash to output the date format in epoch_millis format, and all the documents are discarded when written to that index. If I try to write to a different index with a mapping of strict_date_optional_time then all works perfectly, but the documents are stored in the standard format e.g. "2017-10-24T08:26:09.000Z"

Is there any way to get logstash to output a date in epoch_millis format for compatibility?

I've tried https://github.com/wiibaa/logstash-filter-date_formatter but Joda Time doesn't seem to support Unix timestamps.

Any pointers here? Thanks!

Edit:

I've tried using Ruby to set @timestamp to this, but that results in an error:
event.set('@timestamp',event.get('@ts').to_i)
Ruby exception occurred: wrong argument type Fixnum (expected LogStash::Timestamp)

But it looks like I can set @ts to to this
event.set('@ts',event.get('@ts').to_i)

Guessing @timestamp here is a reserved name and can't be overridden.

Still doesn't solve the problem though as something is casting the integer timestamp to a date somewhere.

Think I'm getting confused here now with @ts and @timestamp, and a combination of issues (notably overwriting the type of @timestamp). I now have @ts stored as an integer using the Ruby code above. Maybe there is a better way to do it?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.