Help with parse logs not standardized

Elastic Stack Logstash
1

Hi,
I'm starting my adventure with Logstash (ELK). I would like parse logs:

2020-02-11 15:10:17.435 DEV_CTRL UPPER OUTPUT/ue/ue1/dev_ctrl/logger IMSI=01723600 Sending message BearerContextReleased: BearerContextReleased: imsi=01723600
2020-02-11 15:10:17.436 EVENT           /ue/ue1/logger          IMSI=01723600 Send delete default bearer for imsi: 01723600 has finished
2020-02-11 15:10:17.436 NAS STATE       /ue/ue1/nas/logger      IMSI=01723600 EpsBearerId=EPS51 From: BEARER CONTEXT ACTIVE To: BEARER CONTEXT INACTIVE
2020-02-11 15:10:17.436 INTERNAL LOWER OUTPUT/ue/ue1/internal/logger IMSI=0101723600 Sending message DeleteDefaultBearer: primitiveIdentity {
2020-02-11 15:10:17.405 LOWER INPUT/logger IMSI=0101723601 LowerUeId=4 PCI=18 CELL=cell1 Receiving message CRLC_RB_CONFIG_CNF
  pid: DeleteDefaultBearer
}
deleteDefaultBearer {
  ueId {
    ueId: 58
    imsi: 2101723600
  }
  epsBearerId: 51
}

I started with Logstash and grok filter. It is a very difficult to build grok pattern to match all logs.
When I created grok pattern for one line: %{TIMESTAMP_ISO8601}%{SPACE}%{WORD:Layer}%{SPACE}%{WORD:From}%{SPACE}%{WORD:Where}\s*%{PATH}%{SPACE}%{WORD:IMSI}=%{INT:Imsi}%{SPACE}%{WORD:Action}%{SPACE}%{WORD:what}%{SPACE}%{WORD:Message}:\s+%{DATA:Message_content}.

In logs I have got information:
[2020-03-14T13:08:27,577][WARN ][logstash.filters.grok ] Timeout executing grok '%{TIMESTAMP_ISO8601}%{SPACE}%{WORD:Layer}%{SPACE}%{WORD:From}%{SPACE}%{WORD:Where}\s*%{PATH}%{SPACE}%{WORD:IMSI}=%{INT:Imsi}%{SPACE}%{WORD:Action}%{SPACE}%{WORD:what}%{SPACE}%{WORD:Message}:\s+%{DATA:Message_content}' against field 'message' with value

Can you advise me how to parse this kind of logos? I tried also with Dissect filter but so far I have a problem with start logstash with this filter in docker.

Regards,
Ernest

2

UNIXPATH (which is part of PATH) is extremely expensive when it does not match. At the very least you should anchor your patten to the start of line using ^.

3

Hi,
I changed PATH on regex expression and add ^ on the beginning of line . Now I do not have got error logs. Thank you.

4

Hi Ernest,

if Badger's answer sorted out your problem, please set it as a solution. That way future readers will know this thread has been solved.

Thanks

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.