Logviewing - Filebeat

Elastic Stack Beats
1

Hello everyone,

Hope you are doing well! I am exploring the possibilities of log viewing through Kibana. I am new to ELK so pardon me for asking dumb questions.
I am using version 7.9.2 for ELK and filebeat as well. so I am sending logs through filebeat directly to Elasticsearch.

now I have multiline logs and following is the specific format of logs.

Trace: 2021/03/08 11:12:44.749 02 t=9CFE88 c=UNK key=P8 tag= (13007004)
  SourceId: com.ibm.ejs.ras.ManagerAdmin
  ExtendedMessage:  TRAS0018I: The trace state has changed. The new trace state is *=info.

but I want to use date and time from my logs instead of the timestamp Kibana provides which is the timestamp when logs are processed. Right now I am using dissect processor and the configuration for that is below.

processors:
   - dissect:
      tokenizer: '%{}: %{DATE} %{TIME} %{}' 
      field: "message"
      target_prefix: ""

but there is something wrong with the configuration because it's not reading proper logs as you can see below.

@timestamp: Apr 20, 2021 @ 14:16:52.274	
DATE: Mar 7, 2021 @ 19:00:00.000	
TIME: 11:12:44.733	
_id: DbR_8HgBNcW2t3BO5hep	
_index: testdata-7.9.2-2021.04.20	
_score: - 	
_type: _doc	
agent.ephemeral_id: 21930328-1007-4a77-9c84-a670785f6456	
agent.hostname: DESKTOP-RP62F3S	
agent.id: cc160ad5-eedb-401d-b9da-9a5b65d350a4	
agent.name: DESKTOP-RP62F3S	
agent.type: filebeat	
agent.version: 7.9.2	
ecs.version: 1.5.0
host.name: DESKTOP-RP62F3S
input.type: log
log.file.path: C:\ELK\Logs\newfile.log
log.offset: 0	
message : Trace: 2021/03/08 11:12:44.733 02 t=9CFE88 c=UNK key=P8 tag= (13007004)

now as you can see from the message, it is reading wrong date but correct time. I am not sure why.

I have two questions here

  1. is there anything wrong with my dissect processor configuration as it is reading wrong date? and while creating index it gives me two fields to choose @Timestamp and DATE. I selected date. is there any way to just override @timestamp with dissect processor?

  2. is there any way through dissect processor to handle multiline logs. I mean I know I can put if statement to see it as a new log when it sees "TRACE" keyword. can that work?

I hope I was able to explain my problem properly. any help would be much appreciated.

Thanks,
Akhil

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.