In order to convert the epoch timestamp (e.g. 1680940932415) to date and time format (e.g. '2006-01-02 15:04:05'), you need to make a few changes to your configuration file: Make sure that you specify the correct field name containing the timestamp. In your case, it is the "timestamp" field. Specify the correct format for the timestamp you want to convert. For your timestamp, the format must be "unix_ms". Specify the desired date and time output format with the "layout" parameter. In your case it should be '2006-01-02 15:04:05'.
I apologize for the confusion in the previous response. To solve this problem, you can use the Go time package to convert the epoch timestamp to a valid timestamp format that can be analyzed by the timestamp processor. Here is an example of how to do this in Go: package main
In this example, we first convert the epoch timestamp to a Unix timestamp by dividing it by 1000 and getting the remainder using the module operator. Then we create a time object using the Unix function and pass the converted Unix timestamp and the remaining milliseconds. Finally, we format the converted time using the format string "2006-01-02T15:04:05.000Z07:00", which is a valid format that can be analyzed by the timestamp processor.
i looked at various reponses on similar queries, but couldnt make them work.
so i decided to do this filtering in ingest pipeline of Elasticsearch
Here's the pipeline that i used to convert epoch data to date time field in Elasticsearch