PubSub Logstash input stuck on authentication


(Amr Motan) #1

Hi,

I am trying to pull logs from GCP Stackdriver logs through pubsub. I followed the instructions in the plugin documentation, I created the topic and the subscriber and made sure that I am authenticated to access the GCP project. Check the attached image for the configurations I used:

When I run logstash the following is printed in the logs 10s of times:

and before closing the pipeline, the following logs appear:

[2018-02-07T09:27:21,608][DEBUG][logstash.pipeline        ] Pipeline has been shutdown {:pipeline_id=>"main", :thread=>"#<Thread:0x6aab64a2@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}
[2018-02-07T09:27:31,321][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: java.nio.channels.ClosedSelectorException
[2018-02-07T09:27:31,321][DEBUG][logstash.agent           ] sun.nio.ch.SelectorImpl.keys(SelectorImpl.java:68)
org.jruby.util.io.SelectorPool.put(SelectorPool.java:88)
org.jruby.util.io.SelectExecutor.selectEnd(SelectExecutor.java:59)
org.jruby.util.io.SelectExecutor.go(SelectExecutor.java:44)
org.jruby.RubyIO.select(RubyIO.java:3405)
org.jruby.RubyIO$INVOKER$s$0$3$select.call(RubyIO$INVOKER$s$0$3$select.gen)
org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:721)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200)
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:314)
org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:77)
org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:192)
org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:131)
org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:339)
org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:73)
org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:132)
org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:148)
org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:73)
org.jruby.runtime.Block.call(Block.java:124)
org.jruby.RubyProc.call(RubyProc.java:289)
org.jruby.RubyProc.call(RubyProc.java:246)
org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:104)
java.lang.Thread.run(Thread.java:748)
[2018-02-07T09:27:31,321][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,321][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] Error in select: closed stream (IOError)
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,322][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,323][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,323][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'
[2018-02-07T09:27:31,323][DEBUG][logstash.agent           ] ["org/jruby/RubyIO.java:3405:in `select'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/reactor.rb:29:in `run_internal'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/reactor.rb:138:in `block in run_in_thread'"]
[2018-02-07T09:27:31,323][DEBUG][logstash.agent           ] 2018-02-07 09:27:31 +0000: Listen loop error: #<IOError: closed stream>
[2018-02-07T09:27:31,324][DEBUG][logstash.agent           ] org/jruby/RubyIO.java:3405:in `select'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:322:in `handle_servers'
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/puma-2.16.0-java/lib/puma/server.rb:296:in `block in run'

Any ideas What am I missing?


(Amr Motan) #2

I found that I was failing on authentication. I checked the code of the plugin and I think that these logs are misleading (talking about Authorizing... Authorized).

If someone can modify the logs to be more meaningful that would be awesome.

The problem was that I provided the topic and subscription names with the full path. But I found that in the code the author did that and all that you have to do is providing the customized name you set for your topic/subscription.

Regards


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.