5.1.1: Plugin installation behind Proxy


#1

Hi,
I'm finding that plugin installation is failing from behind proxy. This has me wondering how proxying should be configured in the current version. Neither of these approaches are working for me:

# export HTTP_PROXY=http://<proxy_host>:<proxy_port>
# export HTTPS_PROXY=https://<proxy_host>:<proxy_port>
# /usr/share/logstash/bin/logstash-plugin install logstash-filter-zeromq

...or add jvm args to jam.options:

-DproxyHost=<proxy_host>
-DproxyPort=<proxy_port>

Neither works, generating this output instead:

ERROR: Something went wrong when installing logstash-input-zeromq, message: Connection refused - Connection refused

However, if I test internet connectivity from the command-line it works:

# wget --spider http://example.com
Spider mode enabled. Check if remote file exists.
--2017-01-03 15:05:18--  http://example.com/
Resolving proxy.ocado.com (proxy.ocado.com)... 192.168.253.50
Connecting to proxy.ocado.com (proxy.ocado.com)|192.168.253.50|:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: 606 [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving

Would appreciate feedback/help here...

Regards,
David


(Mark Walkom) #2

Please format your code with the </> button, it’s very hard to read otherwise.


#3

Ok, done that...


(Joern Ott) #4

I am encountering the same issue here in a docker container where the entrypoint script ensures that certain plugins are installed prior to starting elasticsearch. This used to work end of November and now fails. As I am using the same version of Elasticsearch, I assume that it is not a change in the logstash-plugin code but something in the plugin repository changed which breaks our proxy connection.


(Sandro Scanu) #5

Same issue here, just looking for the plugins to download an install it with bin/logstash-plugin install --local {plugin} but cannot find the plugins.
I'm also behind a proxy. bin/logstash-plugin list works...


(Roland L) #6

Same problem here. I'm using Logstash 5.1.1 and have set the proxy according to: https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#proxy-plugins:

root@server:/usr/share/logstash/bin# echo $http_proxy
http://proxy.host:3128
root@server:/usr/share/logstash/bin# echo $https_proxy
http://proxy.host:3128
root@server:/usr/share/logstash/bin# echo $HTTP_PROXY
http://proxy.host:3128
root@server:/usr/share/logstash/bin# echo $HTTPS_PROXY
http://proxy.host:3128

Trying to install the "multiline" plugin and I get a Connection refused:

root@server:/usr/share/logstash/bin# DEBUG=1 ./logstash-plugin install logstash-filter-multiline
DEBUG: exec /usr/share/logstash/vendor/jruby/bin/jruby --1.9 /usr/share/logstash/lib/pluginmanager/main.rb install logstash-filter-multiline
Looking if package named: logstash-filter-multiline exists at https://artifacts.elastic.co/downloads/logstash-plugins/logstash-filter-multiline/logstash-filter-multiline-5.1.1.zip
Errno::ECONNREFUSED: Connection refused - Connection refused
          initialize at org/jruby/ext/socket/RubyTCPSocket.java:126
                open at org/jruby/RubyIO.java:1197
             connect at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763
             timeout at org/jruby/ext/timeout/Timeout.java:98
             connect at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:763
            do_start at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:756
               start at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:745
               start at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/net/http.rb:557
               start at /usr/share/logstash/lib/pluginmanager/utils/http_client.rb:13
  remote_file_exist? at /usr/share/logstash/lib/pluginmanager/utils/http_client.rb:23
   get_installer_for at /usr/share/logstash/lib/pluginmanager/pack_fetch_strategy/repository.rb:27
              create at /usr/share/logstash/lib/pluginmanager/install_strategy_factory.rb:15
                each at org/jruby/RubyArray.java:1613
              create at /usr/share/logstash/lib/pluginmanager/install_strategy_factory.rb:14
             execute at /usr/share/logstash/lib/pluginmanager/install.rb:27
                 run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
             execute at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/subcommand/execution.rb:11
                 run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67
                 run at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132
              (root) at /usr/share/logstash/lib/pluginmanager/main.rb:46
root@server:/usr/share/logstash/bin#

It seems that logstash-plugin is simply ignoring the proxy.


(Joern Ott) #7

I accidentally installed the latest logstash version instead of the intended version. When I fixed that (thus reverting to 5.0.1), the plugin installation works again. I also had to set not only HTTP_PROXY but also HTTPS_PROXY for all plugins to succeed.

This looks like a regression to me (as it worked in 5.1.1) but I am not deep enough into logstash to determine whether this is due to plugin-install or the bundled jruby version.


Error Installing plugin
(Mark Walkom) #8

See https://github.com/elastic/logstash/issues/6044 for the problem and some suggested workarounds.

This will be fixed in 5.2 though! :slight_smile:


(Roland L) #9

Thanks @warkolm! At least the download seems to work now, using:

export JRUBY_OPTS="-J-Dhttp.proxyHost=proxy.host -J-Dhttp.proxyPort=3128

Unfortunately I get a different error this time:

root@server:/usr/share/logstash/bin# ./logstash-plugin install logstash-filter-multiline
Validating logstash-filter-multiline
Installing logstash-filter-multiline
Error Bundler::InstallError, retrying 1/10
An error occurred while installing logstash-core-event-java (5.1.1), and Bundler cannot continue.
Make sure that `gem install logstash-core-event-java -v '5.1.1'` succeeds before bundling.
WARNING: SSLSocket#session= is not supported
^Croot@server:/usr/share/logstash/bin#

And if I try to use the suggested command, I get:

root@server:/usr/share/logstash/bin# gem install logstash-core-event-java -v '5.1.1'
ERROR:  Could not find a valid gem 'logstash-core-event-java' (= 5.1.1) in any repository

(Eunseon Song) #10

Hi,
I finally solved this issue!! i hope it helps you :smiley:

  1. remove and re-install logstash-5.1.1 (start with pure package)
  2. use follow script

(Roland L) #11

Thanks! That works!

So it seems the only thing I was missing was JARS_SKIP='true' - whatever that means...

Conclusion for 5.1.1:

  1. you have to set the JRUBY_OPTS for logstah-plugin to use the proxy
  2. you have to set http_proxy and https_proxy for Ruby Gem (which is apparently called by logstash-plugin)
  3. and you have to set JARS_SKIP='true' also (for Ruby Gem?) for the installation to work

What a mess...


#12

Can I have a definitive statement on the fix for this please? I've upgraded to 5.2.0 and i still cannot install plugins. And, the above workaround fails too:

# bin/logstash-plugin install logstash-filter-de_dot
Validating logstash-filter-de_dot
Installing logstash-filter-de_dot
Error Bundler::InstallError, retrying 1/10
An error occurred while installing logstash-core-event-java (5.2.0), and Bundler cannot continue.
Make sure that `gem install logstash-core-event-java -v '5.2.0'` succeeds before bundling.
WARNING: SSLSocket#session= is not supported

Regards,
David


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.