Help with parse logs not standardized

I'm starting my adventure with Logstash (ELK). I would like parse logs:

2020-02-11 15:10:17.435 DEV_CTRL UPPER OUTPUT/ue/ue1/dev_ctrl/logger IMSI=01723600 Sending message BearerContextReleased: BearerContextReleased: imsi=01723600
2020-02-11 15:10:17.436 EVENT           /ue/ue1/logger          IMSI=01723600 Send delete default bearer for imsi: 01723600 has finished
2020-02-11 15:10:17.436 NAS STATE       /ue/ue1/nas/logger      IMSI=01723600 EpsBearerId=EPS51 From: BEARER CONTEXT ACTIVE To: BEARER CONTEXT INACTIVE
2020-02-11 15:10:17.436 INTERNAL LOWER OUTPUT/ue/ue1/internal/logger IMSI=0101723600 Sending message DeleteDefaultBearer: primitiveIdentity {
2020-02-11 15:10:17.405 LOWER INPUT/logger IMSI=0101723601 LowerUeId=4 PCI=18 CELL=cell1 Receiving message CRLC_RB_CONFIG_CNF
  pid: DeleteDefaultBearer
deleteDefaultBearer {
  ueId {
    ueId: 58
    imsi: 2101723600
  epsBearerId: 51

I started with Logstash and grok filter. It is a very difficult to build grok pattern to match all logs.
When I created grok pattern for one line: %{TIMESTAMP_ISO8601}%{SPACE}%{WORD:Layer}%{SPACE}%{WORD:From}%{SPACE}%{WORD:Where}\s*%{PATH}%{SPACE}%{WORD:IMSI}=%{INT:Imsi}%{SPACE}%{WORD:Action}%{SPACE}%{WORD:what}%{SPACE}%{WORD:Message}:\s+%{DATA:Message_content}.

In logs I have got information:
[2020-03-14T13:08:27,577][WARN ][logstash.filters.grok ] Timeout executing grok '%{TIMESTAMP_ISO8601}%{SPACE}%{WORD:Layer}%{SPACE}%{WORD:From}%{SPACE}%{WORD:Where}\s*%{PATH}%{SPACE}%{WORD:IMSI}=%{INT:Imsi}%{SPACE}%{WORD:Action}%{SPACE}%{WORD:what}%{SPACE}%{WORD:Message}:\s+%{DATA:Message_content}' against field 'message' with value

Can you advise me how to parse this kind of logos? I tried also with Dissect filter but so far I have a problem with start logstash with this filter in docker.


UNIXPATH (which is part of PATH) is extremely expensive when it does not match. At the very least you should anchor your patten to the start of line using ^.

I changed PATH on regex expression and add ^ on the beginning of line . Now I do not have got error logs. Thank you.

Hi Ernest,

if Badger's answer sorted out your problem, please set it as a solution. That way future readers will know this thread has been solved.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.