Logstash 8.1, Protobuf error - add_file

Long time user here, love Elasticstack but cant get over this problem. Trying to create input for protobuf. Input stanza below. LS stops after throwing java exception, cant get pass this
-tried to get this to work in 7.x, no love
-upgraded to 8.1 yesterday to see if this solves problem but still exists.
-tried ruby comipled rb file and then google compiled rb file.
-tried different options, but no effect
-latest try today with same results as 7.x
Searched online for solution, usually can overcome these with Google fu, but protobuf codec doesn't seem to have a lot of examples, besides kafka and even that is replicated on different sites.
Details are;
Centos7
uname -a
Linux 3.10.0-1160.42.2.el7.x86_64 #1 SMP Tue Sep 7 14:49:57 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

java --version
openjdk 17.0.1 2021-10-19
OpenJDK Runtime Environment 21.9 (build 17.0.1+12)
OpenJDK 64-Bit Server VM 21.9 (build 17.0.1+12, mixed mode, sharing)

yum info Elasticsearch logstash kibana

Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile

Name : kibana
Arch : x86_64
Version : 8.0.1
Release : 1
Size : 726 M
Repo : installed
From repo : Elasticsearch-8.x
Summary : Explore and visualize your Elasticsearch data
URL : https://www.elastic.co
License : Elastic License
Description : Explore and visualize your Elasticsearch data

Name : logstash
Arch : x86_64
Epoch : 1
Version : 8.0.1
Release : 1
Size : 601 M
Repo : installed
From repo : Elasticsearch-8.x
Summary : An extensible logging pipeline
URL : Logstash: Collect, Parse, Transform Logs | Elastic
License : Elastic License
Description : An extensible logging pipeline

Config snippet of logstash input stanza

   host => "0.0.0.0" # default: 0.0.0.0
   port => 9280 # default: 8080
   tags => "PB_analytics"
   codec => protobuf
        {
         class_name => "com.ramp.analytics.proto.PacketWrapper"
         class_file => '/usr/share/logstash/protobuf/analytics_pb-3-5-0.rb'
         protobuf_root_directory => "/usr/share/logstash/protobuf/"
         protobuf_version => 3
        }
   threads => 4
   type => "protobuf_http"
   }```

/usr/share/logstash/bin/logstash-plugin list --verbose | grep protobuf
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-protobuf (1.2.5)

ERROR in /var/log/logstash/logstash-plain.log
```[2022-03-05T11:30:55,049][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.0.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-03-05T11:30:55,051][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-03-05T11:30:56,431][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-03-05T11:30:57,955][INFO ][org.reflections.Reflections] Reflections took 74 ms to scan 1 urls, producing 120 keys and 417 values
[2022-03-05T11:30:59,082][ERROR][logstash.codecs.protobuf ] Unable to load file: /usr/share/logstash/protobuf/analytics_pb-3-5-0.rb. Reason: #<NoMethodError: undefined method `add_file' for #<Google::Protobuf::Builder:0x546cecd3>>
[2022-03-05T11:30:59,085][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (NoMethodError) undefined method `add_file' for #<Google::Protobuf::Builder:0x546cecd3>", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:85)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:376)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
[2022-03-05T11:30:59,094][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[2022-03-05T11:30:59,107][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}
[2022-03-05T11:30:59,114][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit```

Ruby file complied with Google complier 3.5.0

The codec is getting an exception here when it tries to load that file. I do not understand what the error means.

1 Like

The only thing I found online is from this link here,
from 2019
This error is caused by an incompatibility between your protoc version (probably 3.7?) and the rather old jruby implementation that is shipped with this codec. The problem is that the jruby version is not well maintained and we cannot easily update it ourselves. We're looking into this though.
In the meantime you can workaround this by using the 3.5.0 version of protoc.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.