Timestamp - Filebeat

Elastic Stack Beats
1

Hello everyone,

Hope you are doing well! I am exploring the possibilities of log viewing through Kibana. I am using version 7.9.2 for ELK and filebeat as well. so I am sending logs through filebeat directly to Elasticsearch.

now I have multiline logs and following is the specific format of logs.

Trace: 2021/03/08 11:12:44.749 02 t=9CFE88 c=UNK key=P8 tag= (13007004)
  SourceId: com.ibm.ejs.ras.ManagerAdmin
  ExtendedMessage:  TRAS0018I: The trace state has changed. The new trace state is *=info.

but I want to use date and time from my logs instead of the timestamp Kibana provides which is the timestamp when logs are processed. Right now I am using dissect processor and the configuration for that is below.

processors:
   - dissect:
      tokenizer: '%{}: %{+DATE} %{+DATE} %{}' 
      field: "message"
      target_prefix: ""

processors:
  - timestamp:
      field: DATE
      layouts:
         - '2021/03/08 11:12:44.749'
  - drop_fields:
      fields: [DATE]

I have two questions here

  1. here, I think I am on the right way but I am not sure what am i supposed to put in layouts field so I have put time format from my logs. but it is not working.
    can anyone tell me what am i supposed to do here?
  2. is there any way through dissect processor to handle multiline logs. I mean I know I can put if statement to see it as a new log when it sees "TRACE" keyword. can that work?

I hope I was able to explain my problem properly. any help would be much appreciated.

Thanks,
Akhil

2

So look here for the timestamp processor docs, Timestamp | Filebeat Reference [7.12] | Elastic, it looks like u have the right format. Why do u have 2 processors: keys, u should only have one.

3

And look at Dissect combined with multiline pattern gives errors - #2 by shaunak to see how to do multi line dissect.

4

Thank you Alex for the right direction. I tried this but it is not working. I tried everything that's why I put this issue here as I needed fresh pair of eyes on this one. may be i missed something. can you see anything wrong apart from the two processor thing. your help would be much appreciated.

5

I don't see anything that jumps out at me. Though check the docs on the dissect, when combining words using %{+...}, idk if the first value needs the +.

6

Hi Alex,

Actually my logs are in following format. I needed both date and time so I used + to extract both as a date.

Trace: 2021/03/08 11:12:44.733 02 t=9CFE88 c=UNK key=P8 tag= (13007004)

Thanks,
Akhil

7

Ya i had to check the dissect docs to correct myself. Have you remove the multiple processors keys?? you should only have 1.

8

yes, I removed two processors. now there is only one. it is fixed now. it was a problem with layout. Thanks for helping me Alex.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.