TCP output plugin blocking port across configuration reload


(vinayak shukre) #1

Hello Logstash community members,
I am relatively new to logstash and want to read data sent by TCP output plugin in my java code. My conf file is shown below.

input
{
        tcp
        {
                port => 5518
                type => "syslogType"
        }
}
filter{
      grok{
  // some grok is here.
	}
}
output{
	tcp
	{
		host => "localhost"
		port => 5519
		mode => "server"
		codec => "json_lines"
	}
file {
   path => "/tmp/preview.txt"
   codec => "json"
}
}

This works fine but I also have a requirement to change the grok written in the conf file without logstash restart. To achieve this I edit conf file on the fly using vi editor. I have config.reload.automatic: true and have 2 pipelines defined in pipelines.yml in logstash version 6.5.3 OSS on my Ubuntu machine. When Logstash tries to reload the configuration, I get below error.

Jan 10 17:30:57 my-pc logstash[30898]: [2019-01-10T17:30:57,413][INFO ][logstash.pipelineaction.reload] Reloading pipeline {"pipeline.id"=>:preview_pipeline}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,493][WARN ][org.logstash.execution.ShutdownWatcherExt] {"inflight_count"=>0, "stalling_threads_info"=>{}}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,501][ERROR][org.logstash.execution.ShutdownWatcherExt] The shutdown process appears to be stalled due to busy or blocked plugins. Check the logs for more information.
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,802][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"preview_pipeline", :thread=>"#<Thread:0x73a79803 run>"}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,925][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"preview_pipeline", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,932][INFO ][logstash.outputs.tcp     ] Starting tcp output listener {:address=>"localhost:5519"}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,936][ERROR][logstash.outputs.tcp     ] Could not start TCP server: Address in use {:host=>"localhost", :port=>5519}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,944][ERROR][logstash.pipeline        ] Error registering plugin {:pipeline_id=>"preview_pipeline", :plugin=>"#<LogStash::OutputDelegator:0x37af5e47>", :error=>"Address already in use - bind - Address already in use", :thread=>"#<Thread:0x62072031 run>"}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,949][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"preview_pipeline", :exception=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:127:in `initialize'", "org/jruby/RubyIO.java:875:in `new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-tcp-5.0.3/lib/logstash/outputs/tcp.rb:115:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:102:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:46:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:594:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x62072031 run>"}
Jan 10 17:31:02 my-pc logstash[30898]: [2019-01-10T17:31:02,961][ERROR][logstash.agent           ] Failed to execute action {:id=>:preview_pipeline, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<preview_pipeline>, action_result: false", :backtrace=>nil}

Can someone please help me identify why on reloading configuration logstash does not release the port 5519 which is my output port ? If I stop the logstash service and start it again then everything works fine. I want same to happen with conf reload. Please help. Thanks.


#2

Feels like a bug to me. Looking at the code, it never closes a server socket. So if it restarts the previous instance is still listening.


(vinayak shukre) #3

Thanks for your reply. Good to know that this is bug.


(system) closed #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.