Logstash grok extract error messages from nifi logs

[databeata]

How do I extract error messages from nifi logs?

log 1:

2018-07-30 18:43:32,656 ERROR [Timer-Driven Process Thread-8] o.a.n.p.a.storage.ListAzureBlobStorage ListAzureBlobStorage[id=1c111111-0164-1000-ca9e-6d37fd307870] Failed to perform listing on remote host due to java.io.IOException: com.microsoft.azure.storage.StorageException: The specified container does not exist.: {}

extract 1:

com.microsoft.azure.storage.StorageException

log 2:

2018-07-30 17:22:13,282 ERROR [Timer-Driven Process Thread-8] o.a.n.p.a.storage.PutAzureBlobStorage PutAzureBlobStorage[id=ae11111b-3cfa-1cba-ce19-c38a228875a9] Failed to put Azure blob : java.lang.IllegalArgumentException: The argument must not be null or an empty string. Argument name: blobName.

extract 2:

java.lang.IllegalArgumentException

This is my grok now.

grok {
match => {
"message" => "%{NOTSPACE:log_date} %{NOTSPACE:log_time} %{NOTSPACE:log_level} %{GREEDYDATA:log_text}"
}
}

[Badger]

That looks like a very reasonable initial pattern. What do you not like about the results?

[databeata]

I dont get error message categories like "com.microsoft.azure.storage.StorageException", "java.lang.IllegalArgumentException" etc. in a separate field because I have {GREEDYDATA:log_text}.

[Badger]

You could pull some additional fields out if it is useful using

    grok { match => { "message" => [ "%{NOTSPACE:log_date} %{NOTSPACE:log_time} %{NOTSPACE:log_level} \[(?<thread>[^\]]+)\] %{NOTSPACE:classname} %{WORD:class}\[(?<id>[^\]]+)\] %{GREEDYDATA:log_text}" ] } }

Then pick out the last thing that looks like a class name in log_text using

    grok { match => { "log_text" => [ "%{HOSTNAME}: %{HOSTNAME:exception}", "%{HOSTNAME:exception}" ] } }