Multiline codec not picking up custom patterns

With this config:

input {
   file {
      codec => multiline {
        ...
        patterns_dir => [
          '/path/to/patterns'
        ]
        ...
      }
   }
}

and the existing / readable file:

/path/to/patterns/file

that contains user patterns, I'm getting this:

[DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/path/to/patterns"]}

The patterns file does not get picked up at all, everything else is from logstash-patterns-core-4.1.2:

[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/haproxy"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/bro"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/exim"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/aws"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/bacula"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/postgresql"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/redis"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/squid"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/nagios"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/mcollective"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/rails"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/ruby"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/firewalls"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/mcollective-patterns"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/mongodb"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/java"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/maven"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/junos"}
[DEBUG][logstash.codecs.multiline] Grok loading patterns from file {:path=>"/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/bind"}

What am I doing wrong?

Rather than deleting your question, it would be better if you could share your solution, as it may help others in the future with a similar problem :slight_smile:

1 Like

@warkolm I deleted it because I changed the config and stopped before figuring out the problem. I wanted to spare others from wasting their time on this, hope that makes sense :slight_smile:

This line was a red herring though:

[DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/path/to/patterns"]}

The above is for config-only loading looks like and doesn't seem to have anything to do with multiline codec, it just happened to be in the same folder as the config, which is what confused me.

Looking at the code nothing more specific is logged in that case, i.e. the path is just ignored if it's not valid for any reason.

So most likely something I had wrong in the original config.

You were missing the trailing / on the value of patterns_dir. It is not optional.

Got it, thanks @Badger - that might have been it then.