Grok pattern file does not exist


I have configured a custom pattern and using patterns_dir to locate it.
But it reports "Grok pattern file does not exist".

root@openapply-prod-app:/etc/logstash/conf.d# ls -la
total 36
drwxrwxr-x 3 root root 4096 Apr 25 09:01 .
drwxrwxr-x 3 root root 4096 Apr 25 08:53 ..
-rw-r--r-- 1 root root   41 Apr 24 11:06 01-beats-input.conf
-rw-r--r-- 1 root root   53 Apr 24 11:06 02-local-syslog-input.conf
-rw-r--r-- 1 root root  536 Apr 24 11:06 10-syslog.conf
-rw-r--r-- 1 root root  112 Apr 24 11:06 11-nginx.conf
-rw-r--r-- 1 root root  605 Apr 25 08:44 16-oa-rails4.conf
-rw-r--r-- 1 root root  541 Apr 24 11:06 30-output.conf
drw-r--r-- 2 root root 4096 Apr 25 08:39 grok_filters

root@openapply-prod-app:/etc/logstash/conf.d# cat 16-oa-rails4.conf 
filter {
  if "oa" in [tags] and "rails4" in [tags] {
    grok {
      patterns_dir => ["/etc/logstash/conf.d/grok_filters"]
      match => {"message" => "%{RAILS4}"}
    if "_grokparsefailure" in [tags] {
      drop { }
    mutate {
      gsub => ["request", "\?\S+", ""]
    if [request_id] {
      mutate {
        gsub => ["request_id", "[\[\]]", ""]
        replace => {"message" => "%{request_id} %{verb} %{request}"}
    } else {
      mutate {
        replace => {"message" => "%{verb} %{request}"}
    date {
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss Z"]

root@openapply-prod-app:/etc/logstash/conf.d# cat grok_filters/rails4 
RCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
RAILS4PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
RAILS4HEAD ^(?:%{RUUID:request_id})?\s*Started %{WORD:verb} "%{URIPATH:request}(?:%{URIPARAM})?" for %{IPORHOST:clientip} at %{RTIME}\n
RPROCESSING (?:%{RUUID})?\s*Processing by %{RCONTROLLER} as (?<format>\S+)\n
RPARAMETERS (?:%{RUUID})?\s*Parameters: %{DATA:params}\n
RAILS4FOOT (?:%{RUUID})?\s*Completed %{NUMBER:response} %{DATA} in %{NUMBER:totalms}ms %{RAILS4PROFILE}

The error:

[2018-04-25T09:22:37,280][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x7d333800 @metric_events_out=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: out value: 0, @metric_events_in=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: in value: 0, @logger=#<LogStash::Logging::Logger:0x28ef49ba @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0xb8cf473>>, @metric_events_time=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: duration_in_millis value: 0, @id=\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x61680916 @metric=#<LogStash::Instrument::Metric:0x5595f3f6 @collector=#<LogStash::Instrument::Collector:0x765d267 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7cf563c0 @store=#<Concurrent::Map:0x00000000067ef0 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x5d74f5f1>, @fast_lookup=#<Concurrent::Map:0x00000000067ef4 entries=117 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events]>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/etc/logstash/conf.d/grok_filters\"], match=>{\"message\"=>\"%{RAILS4}\"}, id=>\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"Grok pattern file does not exist: /etc/logstash/conf.d/grok_filters/rails4"}

It says Grok pattern file does not exist: /etc/logstash/conf.d/grok_filters/rails4, but the file exists, don't know why.

ll /etc/logstash/conf.d/grok_filters/rails4
-rw-r--r-- 1 root root 755 Apr 24 11:06 /etc/logstash/conf.d/grok_filters/rails4

==================== update ==============================
FYI, even setting ownership to logstash on /etc/logstash doesn't solve the problem.
We temporary use root to run Logstash service and it's ok. But I think this is not a good choice.


You seem to have configured the wrong path. The directory exists in /etc/logstash/conf.d/grok_filters while you have specified it as follows:

drw-r--r-- 2 root root 4096 Apr 25 08:39 grok_filters

The directory must be executable. Otherwise nobody can access any files in the directory.

Ah, thank you so much. That makes sense.

I supposed that read permission is enough for this directory. Why the directory must have execute permission? Is there any document? I didn't find it on

Sorry, it's a typo, having fixed.

The root cause is The directory must be executable as @magnusbaeck says.

Why the directory must have execute permission?

Because that's how permissions work on Unix-like systems.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.