RegexpError: undefined

Elastic Stack Logstash
1

I am getting the following error using logstash:8.6.2 docker image:

[2023-05-30T18:42:18,144][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", 
:exception=>#<RegexpError: undefined group option:
 /(?sm)(?<starttime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} |[A-Z][a-z]{2}\s[0-9]{2},\s[0-9]{4}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s(AM|PM)|)(\s|)(?<loglevel>[A-Z]{4,6})(\s|)(?<service>)(\s|)(?<data>\s(.*?)(?=[\r\n]+\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}|\Z))/m>, 
:backtrace=>["org/jruby/RubyRegexp.java:956:in `initialize'", 
"org/jruby/RubyClass.java:897:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:in `compile'", 
"/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:282:in `block in register'", "org/jruby/RubyArray.java:1865:in `each'",
 "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:276:in `block in register'", "org/jruby/RubyHash.java:1519:in `each'",
 "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-grok-4.4.3/lib/logstash/filters/grok.rb:271
:in `register'", 
"org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:75:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:234:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:233:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:601:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:246:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `run'", 
"/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:143:in `block in start'"], "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x6cc5a43f@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}

i am using the following grok regex

(?sm)(?<starttime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} |[A-Z][a-z]{2}\s[0-9]{2},\s[0-9]{4}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s(AM|PM)|)
(\s|)(?<loglevel>[A-Z]{4,6})(\s|)(?<service>)(\s|)
(?<data>\s(.*?)(?=[\r\n]+\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}|\Z))
2

I am using multiple grok entries

  if [type] == "FOO" {
    if [log.file.path] =~ "foo\.log" {
          grok {      
              match => { "message" => "^(?<starttime>)(\s|)(?<service>)(?<endtime>)(\s|)(?<systemid>)(?<loglevel>)(\s|)(?<class>)(\s|)(?<data>.+$)" }
          }
    }
    else if [log.file.path] =~ "bar\.log " {
          grok {      
              match => { "message" => "^(?<starttime>)( |)(?<service>)(?<endtime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} |[A-Z][a-z]{2}\s[0-9]{2},\s[0-9]{4}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s(AM|PM)|)(?<systemid>[a-zA-Z_]+)(\s|)(?<loglevel>\[[A-Z ]{4,}\]|[A-Z:]{4,}|)(\s|)(?<class>(akka|org|com).[a-zA-Z0-9.]+|)(\s|)(?<data>.*$)" }
          }
    }
    else if [log.file.path] =~ "(baz|qux)\.log" {
          grok {      
              match => { "message" => "(?sm)(?<starttime>)( |)(?<service>)(?<endtime>[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3} |[A-Z][a-z]{2}\s[0-9]{2},\s[0-9]{4}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s(AM|PM)|)(?<systemid>[a-zA-Z_]{1,})(\s|)(?<loglevel>\[[A-Z]{4,6}(| )\])(\s|)(?<class>[\w.]{1,})(\s|)(?<data>\s(.*?)(?=[\r\n]+\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}|\Z))" }
          }
    }

  }

  if [type] == "BAR" {
    grok {
        match => { "message" => "^(?<starttime>)" }
    }
  }
3

Why don't you use grok patterns?
It's really difficult to understand like this and there is no a log sample.

[log.file.path] should be: [log][file][path]

4

the docker container fails to start from the get go, and that is the out put i am getting. for the grok pattern, i will check it out, the issue is we have some weird log format.

5

good it be because of this ?

(?=[\r\n]+\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}|\Z))
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.