Reverse proxy for Logstash - is this even possible now?

Hello,
I have Logstash 6.4 installed on Debian 9.5.

Recently I had to move Logstash to one zone where the only option to get out to the internet is over reverse proxy. I have multiple http_pollers in my Logstash configuration, including tweeter plugin, but all of it does not operate because of the connectivity problems.

Config in /etc/profile

MY_PROXY_URL="http://proxy.myproxy.net:8080"
HTTP_PROXY=$MY_PROXY_URL
HTTPS_PROXY=$MY_PROXY_URL
FTP_PROXY=$MY_PROXY_URL
http_proxy=$MY_PROXY_URL
https_proxy=$MY_PROXY_URL
ftp_proxy=$MY_PROXY_URL
export HTTP_PROXY HTTPS_PROXY FTP_PROXY http_proxy https_proxy ftp_proxy
export NODE_EXTRA_CA_CERTS="/etc/ca-certificates/proxy.myproxy.net.crt"

I have specified in the file: /etc/logstash/jvm.options

SET LS_JAVA_OPTS=-Dhttp.proxyHost=http://proxy.myproxy.net -Dhttp.proxyPort=8080

But it seems that Logstash is not using the proxy.

[2018-09-06T12:13:50,054][WARN ][logstash.inputs.twitter ] Twitter client error {:message=>"Connection refused - connect(2) for "199.16.156.20" port 443", :exception=>"Errno::ECONNREFUSED", :backtrace=>["org/jruby/ext/socket/RubyTCPSocket.java:134:in initialize'", "org/jruby/RubyIO.java:875:in new'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter/patches.rb:26:in stream'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter/patches.rb:58:in request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/twitter-5.15.0/lib/twitter/streaming/client.rb:37:in filter'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-twitter-3.0.8/lib/logstash/inputs/twitter.rb:138:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:408:in inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:402:in block in start_input'"], :options=>nil}

Is this even possible to have that configuration working at the current stage of Logstash where a reverse proxy is a must?
If yes, what is the current recommendation about configuring the Logstash behind a reverse proxy?

Thanks!

Solved the problem partially by specifying in the

http_poller configuration

proxy => { host => "proxy.myproxy.net" port => 8080 scheme => 'http' }

and for twitter:

  use_proxy => true
  proxy_address => "172.1.3.10"
  proxy_port => "8080"

However, I am getting: Twitter::Error::GatewayTimeout"
Is there any setting that I can increase the timeout?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.