Logstash Java Errors

HI All: Is anyone familiar with the below logstash error? I am trying to run a basic Combined apache log GROK Filter.

match => { "message" => "%{COMBINEDAPACHELOG}" }
[2024-09-30T20:13:10,962][WARN ][filewatch.readmode.handlers.readfile][main][cc8228215bb7fad837c5662d01d2f6541e2cc120f6d3612b5c8deb4ea23686e8] failed to open {:path=>"/D:/01-evidence/ABZ3604/logs/tomcat8-stderr.2023-06-12.log", :exception=>Java::JavaNioFile::InvalidPathException, :message=>"Illegal char <:> at index 2: /D:/01-evidence/ABZ3604/logs/tomcat8-stderr.2023-06-12.log", :backtrace=>["java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:204)", "java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:175)", "java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)", "java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)", "java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:231)", "org.logstash.filewatch.JrubyFileWatchLibrary$RubyFileExt.open(JrubyFileWatchLibrary.java:103)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.watched_file.RUBY$method$open$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/watched_file.rb:209)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.base.RUBY$method$open_file$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/base.rb:39)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.read_file.RUBY$method$handle_specifically$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/read_file.rb:15)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.base.RUBY$method$handle$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/base.rb:26)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.processor.RUBY$method$read_file$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/processor.rb:21)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.processor.RUBY$block$process_active$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/processor.rb:90)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:151)", "org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:111)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:106)", "org.jruby.runtime.Block.yield(Block.java:189)", "org.jruby.RubyArray.each(RubyArray.java:1981)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:103)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:545)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:76)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:164)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:151)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}
[2024-09-30T20:13:11,821][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

I also am seeing this error:

2024-09-30 20:33:10,936 pool-11-thread-1 ERROR An exception occurred processing Appender plain_rolling org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1019)
        at org.logstash.log.LoggerExt.debug(org/logstash/log/LoggerExt.java:97)
        at D_3a_.Logstash.logstash_minus_core.lib.logstash.instrument.periodic_poller.base.update(D:/Logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:45)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_to(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
        at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1615)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_to(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_observers(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
        at RUBY.timeout_task(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:329)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.safe_task_executor.execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.safe_task_executor.execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.ivar.safe_execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.scheduled_task.process_task(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.timer_set.process_tasks(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.java_executor_service.run(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)
Caused by: org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String

That seems pretty clear to me. What does the configuration of your file input look like?

It looks like this:

input {
  file {
    path => "D:/01-evidence/ABZ3604/logs/*"
    start_position => "beginning"
    #sincedb_path => "/dev/null"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "D:/logstash/logs/logstash-tomcat_logs_read.log"   
  }
}

Which LS version have you downloaded? logstash-8.x-windows-x86_64.zip?

There is the for. slash at the begging of the path.

I am running 8.15.1. I had the / in the beginning. I discovered that the Java errors could be due to another app that was recently installed. I don't think the issue is with the logstash config.

I have little doubt that another app cause issues.
The forward slash in the log, can be by mistake or from LS linux version on Windows.

Have you solved now?

No. the issue remains. I keep getting the same java errors. also when I login to the server before I even start elastic I am getting Java Update Errors.

OK. I have a new server and was able to get it working.

Next Problem. The grok pattern for the tomcat logs is not working.

When I run the grok debugger here are the details:

Sample Data:

192.168.41.10 - - [25/Sep/2024:00:00:01 -0500] "GET / HTTP/1.1" 200 225597

Grok Pattern:


%{IP:client_ip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes}```

I see Structured Data in the below section after hitting simulate.

However:

when I enter that Grok Pattern Logstash errors out and requires me to add extra \ and changes the pattern to beL

"%{IP:client_ip} - - \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes}"

If I dont add the extra \ Logstash errors out. When it runs like this everything gets put into 1 column: event.original.

I suspect something is wrong with the pattern, or I need to adjust something in elasticsearch.

Who can advise how I can proceed?

Ok so I was able to get the Grok pattern to work, however..

now there is a timestamp issue.

The @timestmap column is showing the ingest time and not the time from the log.

Is there an easy fix ?

All good here. I created the column type mapping before ingesting

You can use grok like below. It combines HTTPDUSER and HAproxy patterns, and follow ECS.

input {
  generator { 
    message =>   [ '192.168.41.10 - - [21/Sep/2024:00:00:01 -0500] "GET / HTTP/1.1" 200 225597' ]
    count => 1 } 
}
filter {
  grok { match => { "message" => '%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[user][name]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HAPROXYDATE:[http][request_date]}\] \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int})' } }

  date {
    match => [ "[http][request_date]", "dd/MMM/yyyy:HH:mm:ss ZZ"] 
    target => "@timestamp"
  }
  mutate {  remove_field => ["@version", "event", "host"] }
}
output {
 stdout {codec => rubydebug}
}