Grok pattern creation error

[Alexandros888]

Hello i am trying to make a grok pattern for that type of log:

[2020-05-13T13:31:42,545][DEBUG][logstash.filters.mutate ][filebeat-lasernet] filters/LogStash::Filters::Mutate: removing field {:field=>"message"}

but i receive grok parse failure.

can somebody may help?

[tiagocosta]

Could you share more information about what you are trying to do? Also I've moved the topic to Logstash as I think you will get better support on this matter there.

[Alexandros888]

Hello,

Sorry for the late reply.

I am trying to parse that log:
[2020-05-13T13:31:42,545][DEBUG][logstash.filters.mutate ][filebeat-lasernet] filters/LogStash::Filters::Mutate: removing field {:field=>"message"}

In order to do that i use the following pattern

\[%{TIMESTAMP_ISO8601:logstash.log.timestamp}\]\[%{LOGSTASH_LOGLEVEL:log.level}\s?\]\[%{LOGSTASH_CLASS_MODULE:logstash.log.module}\s*\]\[%{WORD:logstash.log.pipeline_id}\] %{GREEDYMULTILINE:message}

But i receive an error:

How should i change my grok pattern?

[andres-perez (Andrés Pérez)]

I suspect you are using elastic ingest processor instead of logstash, can you clarify this?

You use custom patterns that seem related to ingest processors. I found these after a quick search:

LOGSTASH_CLASS_MODULE [\w\.]+
LOGSTASH_LOGLEVEL INFO|ERROR|DEBUG|FATAL|WARN|TRACE
GREEDYMULTILINE (.|\n)*

The dot notation log.level is not standard for logstash, bracket notation should be instead [log][level].

Anyway, your last pattern doesn't work because WORD pattern doesn't match "filebeat-lasernet". You might use DATA pattern.

A tool like grok constructor is more useful to test patterns as it shows what has and has not matched.

Having said that, if your logs start with a fixed number of bracket fields, it would make more sense to use the dissect filter (for logstash) or dissect processor. It should perform much better and you can forget about patterns.

If you really need to use grok, at least you can start your match patterns with an anchor to the begining of the line, it will perform somewhat better. You can find more information here:

Do you grok Grok?