Hi,
I have configured a custom pattern and using patterns_dir to locate it.
But it reports "Grok pattern file does not exist".
root@openapply-prod-app:/etc/logstash/conf.d# ls -la
total 36
drwxrwxr-x 3 root root 4096 Apr 25 09:01 .
drwxrwxr-x 3 root root 4096 Apr 25 08:53 ..
-rw-r--r-- 1 root root 41 Apr 24 11:06 01-beats-input.conf
-rw-r--r-- 1 root root 53 Apr 24 11:06 02-local-syslog-input.conf
-rw-r--r-- 1 root root 536 Apr 24 11:06 10-syslog.conf
-rw-r--r-- 1 root root 112 Apr 24 11:06 11-nginx.conf
-rw-r--r-- 1 root root 605 Apr 25 08:44 16-oa-rails4.conf
-rw-r--r-- 1 root root 541 Apr 24 11:06 30-output.conf
drw-r--r-- 2 root root 4096 Apr 25 08:39 grok_filters
root@openapply-prod-app:/etc/logstash/conf.d# cat 16-oa-rails4.conf
filter {
if "oa" in [tags] and "rails4" in [tags] {
grok {
patterns_dir => ["/etc/logstash/conf.d/grok_filters"]
match => {"message" => "%{RAILS4}"}
}
if "_grokparsefailure" in [tags] {
drop { }
}
mutate {
gsub => ["request", "\?\S+", ""]
}
if [request_id] {
mutate {
gsub => ["request_id", "[\[\]]", ""]
replace => {"message" => "%{request_id} %{verb} %{request}"}
}
} else {
mutate {
replace => {"message" => "%{verb} %{request}"}
}
}
date {
match => ["timestamp", "yyyy-MM-dd HH:mm:ss Z"]
}
}
}
root@openapply-prod-app:/etc/logstash/conf.d# cat grok_filters/rails4
RUUID \S+
RCONTROLLER (?<controller>[^#]+)#(?<action>\w+)
RTIME (?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND} %{ISO8601_TIMEZONE})
RAILS4PROFILE (?:\(Views: %{NUMBER:viewms}ms \| ActiveRecord: %{NUMBER:activerecordms}ms|\(ActiveRecord: %{NUMBER:activerecordms}ms)?
RAILS4HEAD ^(?:%{RUUID:request_id})?\s*Started %{WORD:verb} "%{URIPATH:request}(?:%{URIPARAM})?" for %{IPORHOST:clientip} at %{RTIME}\n
RPROCESSING (?:%{RUUID})?\s*Processing by %{RCONTROLLER} as (?<format>\S+)\n
RPARAMETERS (?:%{RUUID})?\s*Parameters: %{DATA:params}\n
RAILS4FOOT (?:%{RUUID})?\s*Completed %{NUMBER:response} %{DATA} in %{NUMBER:totalms}ms %{RAILS4PROFILE}
RAILS4 %{RAILS4HEAD}(?:%{RPROCESSING})?(?:%{RPARAMETERS})?%{DATA:context}\n%{RAILS4FOOT}
The error:
[2018-04-25T09:22:37,280][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::FilterDelegator:0x7d333800 @metric_events_out=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: out value: 0, @metric_events_in=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: in value: 0, @logger=#<LogStash::Logging::Logger:0x28ef49ba @logger=#<Java::OrgApacheLoggingLog4jCore::Logger:0xb8cf473>>, @metric_events_time=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events] key: duration_in_millis value: 0, @id=\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", @klass=LogStash::Filters::Grok, @metric_events=#<LogStash::Instrument::NamespacedMetric:0x61680916 @metric=#<LogStash::Instrument::Metric:0x5595f3f6 @collector=#<LogStash::Instrument::Collector:0x765d267 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x7cf563c0 @store=#<Concurrent::Map:0x00000000067ef0 entries=3 default_proc=nil>, @structured_lookup_mutex=#<Mutex:0x5d74f5f1>, @fast_lookup=#<Concurrent::Map:0x00000000067ef4 entries=117 default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :filters, :\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", :events]>, @filter=<LogStash::Filters::Grok patterns_dir=>[\"/etc/logstash/conf.d/grok_filters\"], match=>{\"message\"=>\"%{RAILS4}\"}, id=>\"c7984a2adbaf518d790f55723443e760b1bbb8b6-8\", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>\"*\", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>[\"_grokparsefailure\"], timeout_millis=>30000, tag_on_timeout=>\"_groktimeout\">>", :error=>"Grok pattern file does not exist: /etc/logstash/conf.d/grok_filters/rails4"}
It says Grok pattern file does not exist: /etc/logstash/conf.d/grok_filters/rails4, but the file exists, don't know why.
ll /etc/logstash/conf.d/grok_filters/rails4
-rw-r--r-- 1 root root 755 Apr 24 11:06 /etc/logstash/conf.d/grok_filters/rails4
==================== update ==============================
FYI, even setting ownership to logstash on /etc/logstash doesn't solve the problem.
We temporary use root to run Logstash service and it's ok. But I think this is not a good choice.