Parsing a string delimited log with Dissect

Elastic Stack Logstash
1

Good day all,
I am trying to parse the below log using Dissect.

1541164838.872000[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5

format:
xxxxx.xxxxx[timestamp timestamp] Module: some data Event: some data Info: some data .....

My Filter configuration looks
filter {
dissect {
mapping => {
"message" => "%{SeqNo}[%{date_time} %{+timestamp}] %{Module}Module: %{Event}Event: %{ID}ID: %{Type}Type: %{Count}Count: %{Aff.Obj}Aff.Obj: %{Info}"
}
}
}

When I get it to work, only the timestamp portions and the last element parse correctly. The rest are shifted to the right. ie. Value for Module (1st delimiter) is blank, Event takes up the value for Module and so on.

Any help will be highly appreciated.

Regards,
Lenya

2

From timestamp onwards your patterns are backwards.
Try:

"message" => "%{SeqNo}[%{date_time} %{+timestamp}] Module: %{Module} Event: %{Event} ID: %{ID} Type: %{Type} Count: %{Count} Aff.Obj: %{Aff.Obj} Info: %{Info}"

I find that if I start with the original line as a comment and the mapping then gradually replace the values I need to extract with a %{key} section, then the delimiters are left in the correct places.
1

    # "message" => "1541164838.872000[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"
      "message" => "1541164838.872000[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"

2

      # "message" => "1541164838.872000[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"
      "message" => "%{SeqNo}[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"

3

      # "message" => "1541164838.872000[20181102 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"
      "message" => "%{SeqNo}[%{date_time} 16:20:38.872000] Module: PXYZ-ModuleHandler/8.1/A/1 Event: some event initialized ID: 4800001 Type: 93669594796392449 Count: 7 Aff.Obj: Info: Announcement code list 1 is not initialised! It is used in service class 5"

And so on until the end then I delete the comment.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.