Dates in epoch.milliseconds and ingest node

Elastic Stack Elasticsearch
1

I am using 5.0 alpha5 with an ingest node and I am trying to get a timestamp field that is in epoch.milliseconds (ie. 1473371994.193595) to be formatted as a time field. The field is processed without errors if I allow it to default to type "float." It is actually a date value but if I map the field as date with format "epoch_millis" I get an elasticsearch error "IllegalArguementException[Invalid format: "1.473371994193595E9"] when I try to ingest the data.

I have tried using the ingest node date processor to convert it to date:
{
"date": {
"field": "timestamp",
"target_field": "timestamp",
"formats": ["UNIX_MS"]
}
}
I get the errorIllegalArgumentException{field [timestamp] of type [java.lang.Double] cannot be cast to [java.lang.String]

I have even tried multiplying the value by 1000 using a painless script:
{
"script": {
"lang": "painless",
"inline": "ctx.ts_millis = ctx.timestamp * 1000"
}
}
in the ingest pipeline but I get the same error "IllegalArguementException[Invalid format: "1.473371994193595E12"].

I have also tried converting it to a string but get a similar java.lang casting error.

I have run out of things to try and wonder if there might be someone to provide some new ideas?

2

That's because epoch + milliseconds would only have 3 decimal places. This is more precision than Elasticsearch can handle for date math.

3

Thanks for confirming my suspicions, but my pea-brain hasn't been able to ponder a method to truncate the offending three digits. That's were I was heading with trying to convert the value to string. I even tried integer without success.

4

I don't know if the ingest node can do that, but it's pretty trivial in Logstash.

5

I have a logstash configuration that I'm trying to move away from for efficiency sake. I guess I'll keep it for a while longer.

Thank you for the help.

6

Just providing an update. I located a configuration parameter in the source application that allowed changes from epoch time format to ISO8601. Although the time is still output with microseconds precision (six decimal places after the decimal point), the ingest node is happy to take it and convert it to millisecond precision.

Thanks again for your help. You guys are great.

7

For what it's worth, the doco says the "formats": ["UNIX"] will handle epochs with decimal millis now.
"UNIX - will parse float or int value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001"