Logstash Java Errors

Elastic Stack Logstash
1

HI All: Is anyone familiar with the below logstash error? I am trying to run a basic Combined apache log GROK Filter.

match => { "message" => "%{COMBINEDAPACHELOG}" }

[2024-09-30T20:13:10,962][WARN ][filewatch.readmode.handlers.readfile][main][cc8228215bb7fad837c5662d01d2f6541e2cc120f6d3612b5c8deb4ea23686e8] failed to open {:path=>"/D:/01-evidence/ABZ3604/logs/tomcat8-stderr.2023-06-12.log", :exception=>Java::JavaNioFile::InvalidPathException, :message=>"Illegal char <:> at index 2: /D:/01-evidence/ABZ3604/logs/tomcat8-stderr.2023-06-12.log", :backtrace=>["java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:204)", "java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:175)", "java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)", "java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92)", "java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:231)", "org.logstash.filewatch.JrubyFileWatchLibrary$RubyFileExt.open(JrubyFileWatchLibrary.java:103)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.watched_file.RUBY$method$open$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/watched_file.rb:209)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.base.RUBY$method$open_file$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/base.rb:39)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.read_file.RUBY$method$handle_specifically$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/read_file.rb:15)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.handlers.base.RUBY$method$handle$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/handlers/base.rb:26)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.processor.RUBY$method$read_file$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/processor.rb:21)", "D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_input_minus_file_minus_4_dot_4_dot_6.lib.filewatch.read_mode.processor.RUBY$block$process_active$0(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-file-4.4.6/lib/filewatch/read_mode/processor.rb:90)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:151)", "org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:111)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:106)", "org.jruby.runtime.Block.yield(Block.java:189)", "org.jruby.RubyArray.each(RubyArray.java:1981)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(RubyArray$INVOKER$i$0$0$each.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:561)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:103)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:545)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:76)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:164)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:151)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:212)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:456)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:195)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:346)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:82)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:201)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:188)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:220)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:466)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:244)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}
[2024-09-30T20:13:11,821][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu

I also am seeing this error:

2024-09-30 20:33:10,936 pool-11-thread-1 ERROR An exception occurred processing Appender plain_rolling org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyKernel.inspect(org/jruby/RubyKernel.java:2362)
        at org.jruby.RubyHash.inspect(org/jruby/RubyHash.java:953)
        at org.jruby.RubyHash.to_s(org/jruby/RubyHash.java:1019)
        at org.logstash.log.LoggerExt.debug(org/logstash/log/LoggerExt.java:97)
        at D_3a_.Logstash.logstash_minus_core.lib.logstash.instrument.periodic_poller.base.update(D:/Logstash/logstash-core/lib/logstash/instrument/periodic_poller/base.rb:45)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_to(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:102)
        at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1615)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_to(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:100)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.collection.copy_on_notify_observer_set.notify_observers(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/collection/copy_on_notify_observer_set.rb:64)
        at RUBY.timeout_task(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/timer_task.rb:329)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.safe_task_executor.execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:24)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.safe_task_executor.execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/safe_task_executor.rb:19)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.ivar.safe_execute(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/ivar.rb:169)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.scheduled_task.process_task(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/scheduled_task.rb:285)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.timer_set.process_tasks(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/timer_set.rb:165)
        at D_3a_.Logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.concurrent_minus_ruby_minus_1_dot_1_dot_9.lib.concurrent_minus_ruby.concurrent.executor.java_executor_service.run(D:/Logstash/vendor/bundle/jruby/3.1.0/gems/concurrent-ruby-1.1.9/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79)
Caused by: org.jruby.exceptions.TypeError: (TypeError) no implicit conversion of Hash into String
2

That seems pretty clear to me. What does the configuration of your file input look like?

3

It looks like this:

input {
  file {
    path => "D:/01-evidence/ABZ3604/logs/*"
    start_position => "beginning"
    #sincedb_path => "/dev/null"
    mode => "read"
    file_completed_action => "log"
    file_completed_log_path => "D:/logstash/logs/logstash-tomcat_logs_read.log"   
  }
}
4

Which LS version have you downloaded? logstash-8.x-windows-x86_64.zip?

There is the for. slash at the begging of the path.

5

I am running 8.15.1. I had the / in the beginning. I discovered that the Java errors could be due to another app that was recently installed. I don't think the issue is with the logstash config.

6

I have little doubt that another app cause issues.
The forward slash in the log, can be by mistake or from LS linux version on Windows.

Have you solved now?

7

No. the issue remains. I keep getting the same java errors. also when I login to the server before I even start elastic I am getting Java Update Errors.

8

OK. I have a new server and was able to get it working.

Next Problem. The grok pattern for the tomcat logs is not working.

When I run the grok debugger here are the details:

Sample Data:

192.168.41.10 - - [25/Sep/2024:00:00:01 -0500] "GET / HTTP/1.1" 200 225597

Grok Pattern:


%{IP:client_ip} - - \[%{HTTPDATE:timestamp}\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes}```

I see Structured Data in the below section after hitting simulate.

However:

when I enter that Grok Pattern Logstash errors out and requires me to add extra \ and changes the pattern to beL

"%{IP:client_ip} - - \\[%{HTTPDATE:timestamp}\\] \"%{WORD:method} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:bytes}"

If I dont add the extra \ Logstash errors out. When it runs like this everything gets put into 1 column: event.original.

I suspect something is wrong with the pattern, or I need to adjust something in elasticsearch.

Who can advise how I can proceed?

9

Ok so I was able to get the Grok pattern to work, however..

now there is a timestamp issue.

The @timestmap column is showing the ingest time and not the time from the log.

Is there an easy fix ?

10

All good here. I created the column type mapping before ingesting

11

You can use grok like below. It combines HTTPDUSER and HAproxy patterns, and follow ECS.

input {
  generator { 
    message =>   [ '192.168.41.10 - - [21/Sep/2024:00:00:01 -0500] "GET / HTTP/1.1" 200 225597' ]
    count => 1 } 
}
filter {
  grok { match => { "message" => '%{IPORHOST:[source][address]} (?:-|%{HTTPDUSER:[user][name]}) (?:-|%{HTTPDUSER:[user][name]}) \[%{HAPROXYDATE:[http][request_date]}\] \"(?:%{WORD:[http][request][method]} %{NOTSPACE:[url][original]}(?: HTTP/%{NUMBER:[http][version]})?|%{DATA})\" (?:-|%{INT:[http][response][status_code]:int}) (?:-|%{INT:[http][response][body][bytes]:int})' } }

  date {
    match => [ "[http][request_date]", "dd/MMM/yyyy:HH:mm:ss ZZ"] 
    target => "@timestamp"
  }
  mutate {  remove_field => ["@version", "event", "host"] }
}
output {
 stdout {codec => rubydebug}
}