Long time user here, love Elasticstack but cant get over this problem. Trying to create input for protobuf. Input stanza below. LS stops after throwing java exception, cant get pass this
-tried to get this to work in 7.x, no love
-upgraded to 8.1 yesterday to see if this solves problem but still exists.
-tried ruby comipled rb file and then google compiled rb file.
-tried different options, but no effect
-latest try today with same results as 7.x
Searched online for solution, usually can overcome these with Google fu, but protobuf codec doesn't seem to have a lot of examples, besides kafka and even that is replicated on different sites.
Details are;
Centos7
uname -a
Linux 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
java --version
openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment 21.9 (build 17.0.1+12)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.1+12, mixed mode, sharing)
yum info Elasticsearch logstash kibana
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
- base: mirrors.mit.edu
- epel: mirror.dst.ca
- extras: mirror.centos.iad1.serverforge.org
- updates: mirrors.seas.harvard.edu
Installed Packages
Name : Elasticsearch
Arch : x86_64
Version : 8.0.1
Release : 1
Size : 1.0 G
Repo : installed
From repo : Elasticsearch-8.x
Summary : Distributed RESTful search engine built for the cloud
URL : https://www.elastic.co/
License : Elastic License
Description : Reference documentation can be found at
: Elasticsearch Guide [8.0] | Elastic
: and the 'Elasticsearch: The Definitive Guide' book can be found at
: Elasticsearch: The Definitive Guide [2.x] | Elastic
Name : kibana
Arch : x86_64
Version : 8.0.1
Release : 1
Size : 726 M
Repo : installed
From repo : Elasticsearch-8.x
Summary : Explore and visualize your Elasticsearch data
URL : https://www.elastic.co
License : Elastic License
Description : Explore and visualize your Elasticsearch data
Name : logstash
Arch : x86_64
Epoch : 1
Version : 8.0.1
Release : 1
Size : 601 M
Repo : installed
From repo : Elasticsearch-8.x
Summary : An extensible logging pipeline
URL : Logstash: Collect, Parse, Transform Logs | Elastic
License : Elastic License
Description : An extensible logging pipeline
Config snippet of logstash input stanza
host => "0.0.0.0" # default: 0.0.0.0
port => 9280 # default: 8080
tags => "PB_analytics"
codec => protobuf
{
class_name => "com.ramp.analytics.proto.PacketWrapper"
class_file => '/usr/share/logstash/protobuf/analytics_pb-3-5-0.rb'
protobuf_root_directory => "/usr/share/logstash/protobuf/"
protobuf_version => 3
}
threads => 4
type => "protobuf_http"
}```
/usr/share/logstash/bin/logstash-plugin list --verbose | grep protobuf
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-protobuf (1.2.5)
ERROR in /var/log/logstash/logstash-plain.log
```[2022-03-05T11:30:55,049][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.0.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-03-05T11:30:55,051][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-03-05T11:30:56,431][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-05T11:30:57,955][INFO ][org.reflections.Reflections] Reflections took 74 ms to scan 1 urls, producing 120 keys and 417 values
[2022-03-05T11:30:59,082][ERROR][logstash.codecs.protobuf ] Unable to load file: /usr/share/logstash/protobuf/analytics_pb-3-5-0.rb. Reason: #<NoMethodError: undefined method `add_file' for #<Google::Protobuf::Builder:0x546cecd3>>
[2022-03-05T11:30:59,085][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (NoMethodError) undefined method `add_file' for #<Google::Protobuf::Builder:0x546cecd3>", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:85)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:376)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
[2022-03-05T11:30:59,094][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2022-03-05T11:30:59,107][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}
[2022-03-05T11:30:59,114][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit```
Ruby file complied with Google complier 3.5.0