Timestamp

Elastic Stack Beats
1

Hello everyone,

I am trying to use timestamp from my log file instead of the timestamp when my logs were processed. in order to do that, I am using dissect processor and Timestamp processor together as below.

processors:
  - dissect:
      tokenizer: '%{}: %{DATE} %{DATE} %{}' 
      field: "message"
      target_prefix: ""
      
  - timestamp:
      field: DATE
      layouts:
        - '2021/03/08 11:12:44.749'
  - drop_fields:
      fields: [DATE]

so with dissect processor I am extracting Date from my logs and then with Timestamp processor I am trying to replace DATE with timestamp when logs were processed. but it is not working for some reason even though do not see any errors in filebeat logs but I see that these processors are being used.

2021-04-25T17:56:20.721-0400	DEBUG	[processors]	processors/processor.go:101	Generated new processors: dissect=%{}: %{DATE} %{DATE} %{},field=message,target_prefix=, timestamp=[field=DATE, target_field=@timestamp, timezone=UTC, layouts=[2021/03/08 11:12:44.749]], drop_fields={"Fields":["DATE"],"IgnoreMissing":false}

Following is my log format.

Trace: 2021/03/08 11:12:44.733 02 t=9CFE88 c=UNK key=P8 tag= (13007004)

I am stuck on this from sometime now. I am not sure what is wrong. can anyone please help me?

Thanks.

2

One thing...
Your layout is not correct read this carefully you need to use the reference time for the layout.

The layouts are described using a reference time that is based on this specific time: (i.e you must describe your format as this time)

Mon Jan 2 15:04:05 MST 2006

Since MST is GMT-0700, the reference time is:

01/02 03:04:05PM '06 -0700

To define your own layout, rewrite the reference time in a format that matches the timestamps you expect to parse. For more layout examples and details see the go lang

Something like this...but you will probably need timezone parameter otherwise it will be UTC.

2006/01/02 15:04:05.000

You can also use the test directive to check on startup.

3

Thank you Stephen for the explanation! I will read the documentation carefully and hopefully it works! thanks again

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.