Logstash error after upgrade (of ElasticSearch, Kibana and Logstash) from 7.16.3 to 7.17.29

Elastic Stack Logstash
1

No log ingestion. Logstash wont stay up. Error in logs:

[ERROR][logstash.javapipeline ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{DATAINBRACKETS:log_date} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in block in compile'", "org/jruby/RubyKernel.java:1442:in loop'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in compile'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:282:in block in register'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:276:in block in register'", "org/jruby/RubyHash.java:1415:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:271:in register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in block in register_plugins'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:599:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:245:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:190:in run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:142:in block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x43123674@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:130 run>"}

We tested manual index/shard creation using REST API which worked fine, but think there is some problem with config since the upgrade. Any help or advice welcome.

2

Where in your configuration do you think you have defined the DATAINBRACKETS pattern?

3

This is a custom pattern, is not one of the default GROK patterns from Logstash.

Probably you had this pattern defined on a file inside Logstash installation path, which would change after the upgrade.

You need to recreate this custom pattern and configure your grok filter to use it, it is better to define custom patterns on an external file not part of the installation path.

4

This is resolved, the original relative path meant the patterns file couldnt be found, after the upgrade it seems we needed to provide an absolute path to the patterns file. Thanks for your responses though.