Filebeat timestamp conversion from PST to EST

Input file has PST timestamp like "2020 Feb 12 13:54:07:447 GMT -0800" in a message event like below:
2020 Feb 12 13:54:07:447 GMT -0800 myservice Error [mybusinessworks] BWENGINE-100001 process initialization failed for CommonProcesses/ExceptionHandling/ExcepOverJMSProxy.process

I am trying to convert the Timestamp from PST to EST using the below snippet in Filebeat.

------
  processors:
    - dissect:
        tokenizer: "%{log_time} %{+log_time} %{+log_time} %{+log_time}:%{+log_time}:%{+log_time}:%{+log_time} %{} %{} %{service_name} %{log_level} [%{process_name->}] %{msg}"
        field: "message"
        target_prefix: ""
    - drop_event:
        when:
          not:
            equals:
              log_level: Error
    - timestamp:
        field: log_time
        layouts:
          - '2020 Feb 12 13:54:07:447'
        test:       
          - '2020 Feb 12 13:54:07:447'
    - drop_fields:
        fields: ["log_time","message"] 
------

Error details:

********
c:\Program Files\filebeat>"C:\Program Files\Filebeat\\filebeat.exe" -c "C:\Program Files\Filebeat\\filebeat.yml" -path.home "C:\Program Files\Filebeat" -path.data "C:\\ProgramData\\filebeat" -path.logs "C:\\ProgramData\\filebeat\logs"
Exiting: Failed to start crawler: starting input failed: Error while initializing input: failed parsing time field log_time='2020 Feb 12 13:54:07:447': failed using layout [2020 Feb 12 13:54:07:447] cannot parse [ Feb 12 13:54:07:447] as [0 Feb ]
failed to parse test timestamp
github.com/elastic/beats/v7/libbeat/processors/timestamp.newFromConfig
        /go/src/github.com/elastic/beats/libbeat/processors/timestamp/timestamp.go:79
********

I have removed the GMT part from the time field thinking I can do the conversion easily. but it didn't work. Wondering if I should somehow shuttle the time field to match with one of the layouts like below to convert the time stamp to EST:
layouts:
- '2006-01-02T15:04:05Z'
- '2006-01-02T15:04:05.999Z'
- '2006-01-02T15:04:05.999-07:00'

The default value of the Timezone for the Timestamp processor is UTC. Which is what I want the timestamp value to be converted into.

I am using Filebeat 7.11

does anyone know if there are any additional approved list of layout formats for Timestamp?
I do not know how to convert Jan to "01" and Feb to "02" and so on so that I can reconstruct timestamp in one of the below formats:

  • '2006-01-02T15:04:05Z'
  • '2006-01-02T15:04:05.999Z'
  • '2006-01-02T15:04:05.999-07:00'

I'm not sure I fully understand your use-case here. Elasticsearch only stores data in UTC, so converting to EST doesn't make sense. If you want to convert to UTC, add the timezone field to the timestamp processor, so it knows the time is in PST and to convert it to UTC.

The layout listed on the docs is actually a Golang time thing, and requires those specific time values.
Try changing your layout value to be 2006 Jan 02 15:04:05:999 and see if that works.

The layout

2006 Jan 02 15:04:05:999

did not work

Trying to find find if any other layout would work.

Is it possible to create a new field using a combination of other fields in file beat?

For example,
If I have tokenizer like below:

tokenizer: "%{+log_year} %{+log_month} %{+log_day} %{+log_hour}:%{+log_minutes}:%{+log_seconds}:%{+log_millies} %{+log_timezone} %{+log_lag} %{service_name} %{log_level} [%{process_name->}] %{msg}"

then create a new field (using - add_field processor) by rearranging the above timestamp fields.

The new field that would be created is in one of the acceptable layout formats.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.