Hi all,
I'm issuing a performance problem with grok filter.
I use filebeat 5.4.6 to send log file event to logstash 5.4.6.
I made a very simple grok filter in logstash to extract path and filename of the log file from "source" field from filebeat :
grok {
match => { "source" => "%{UNIXPATH:[filepath]}/%{NOTSPACE:[filename]}" }
}
It works very well with a lot of filename but the filter is very slow when there is many dots, dashs, underscore in the filename.
Example : /var/log/nginx/mynginx01access.log -> very fast
/var/log/nginx/my_nginx-01.access.log -> very slow and CPU costly
I try many pattern to replace %{NOTSPACE } whith %{DATA}, %{GREEDYDATA}... whitout any result. The CPU loads for the filter seems to be an exponential of the number of (.,-,) in the filename.
If you replace (.,-,) whith other special charater (#,$,^,space...), it's fast again.
I don't know how to fix this problem, because I try every possible pattern.
Help would be very appreciated.
Simon