Filebeat timestamp processor to parse from epoch/unix_ms to readable format

Elastic Stack Beats
1

I want to convert an epoch timestamp (ex: 1680940932415) to readable format (ex: '2006-01-02 15:04:05') using timestamp processor (Timestamp | Filebeat Reference [8.7] | Elastic), but am unable to achieve it.

processors:
- decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true
- timestamp:
    field: "timestamp"
    target_field: "newtp"
    layouts:
      - '2006-01-02T15:04:05Z'
      - '2006-01-02T15:04:05.999Z'
      - '2006-01-02T15:04:05.999-07:00'
    test:
      - "1680940932415"

Is this even doable using the timestamp processor? thanks in advance

2

In order to convert the epoch timestamp (e.g. 1680940932415) to date and time format (e.g. '2006-01-02 15:04:05'), you need to make a few changes to your configuration file: Make sure that you specify the correct field name containing the timestamp. In your case, it is the "timestamp" field. Specify the correct format for the timestamp you want to convert. For your timestamp, the format must be "unix_ms". Specify the desired date and time output format with the "layout" parameter. In your case it should be '2006-01-02 15:04:05'.
processors:

  • decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true
  • timestamp:
    field: "timestamp"
    target_field: "newtp"
    test:
    - "1680940932415"
    layouts:
    - "2006-01-02 15:04:05"
    timezone: "UTC"
    locale: "en"

    format: unix_ms

Note that I have added the "timezone" and "locale" parameters to set the timezone and locale to convert the date and time. You can change them according to your requirements

3

Thanks for replying, but this didnt solve my problem. I am still getting the error that the test timestamp cannot be parsed

Exiting: error initializing processors: failed to parse test timestamp: failed parsing time field timestamp='1680940932415'
4

I apologize for the confusion in the previous response. To solve this problem, you can use the Go time package to convert the epoch timestamp to a valid timestamp format that can be analyzed by the timestamp processor. Here is an example of how to do this in Go: package main

import (
"fmt"
"time"
)

func main() {
epochTime := int64(1680940932415)
convertedTime := time.Unix(epochTime/1000, (epochTime%1000)*int64(time.Millisecond))

fmt.Println(convertedTime.Format("2006-01-02T15:04:05.000Z07:00"))

}
In this example, we first convert the epoch timestamp to a Unix timestamp by dividing it by 1000 and getting the remainder using the module operator. Then we create a time object using the Unix function and pass the converted Unix timestamp and the remaining milliseconds. Finally, we format the converted time using the format string "2006-01-02T15:04:05.000Z07:00", which is a valid format that can be analyzed by the timestamp processor.

5

thanks for the reply,

i looked at various reponses on similar queries, but couldnt make them work.
so i decided to do this filtering in ingest pipeline of Elasticsearch
Here's the pipeline that i used to convert epoch data to date time field in Elasticsearch

{
  "epoch-to-date-pipeline": {
    "description": "test pipeline to parse timestamp field",
    "processors": [
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "UNIX_MS"
          ],
          "if": "ctx.timestamp != null && (ctx.timestamp instanceof String || ctx.timestamp instanceof Number)"
        }
      }
    ]
  }
}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.