Building logstash from source fails as the http_proxy env setting is not getting passed in

pkempia · May 16, 2017, 11:33am

I am trying to build logstash 5.3.2 from source. Below are the steps I followed. But the "rake bootstrap" command fails with the "Errno::EHOSTUNREACH: No route to host - connect(2)"

This is been done on a empty docker container hosted on a openstack vm.

Step 1: export the proxy variables
export http_proxy=http://corporate-proxy-ip:port
export https_proxy=https://corporate-proxy-ip:port

Step2 : Get the source file.
wget wget https://github.com/elastic/logstash/archive/v5.3.2.tar.gz

Step3 : yum install ruby

Step4 : gem install rake
Fetching: rake-12.0.0.gem (100%)
Successfully installed rake-12.0.0
Parsing documentation for rake-12.0.0
Installing ri documentation for rake-12.0.0
1 gem installed
Step5 :
bash-4.2# rake bootstrap
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
rake aborted!
Errno::EHOSTUNREACH: No route to host - connect(2)
/home/logstash/rakelib/fetch.rake:54:in download' /home/logstash/rakelib/fetch.rake:11:infetch'
/home/logstash/rakelib/fetch.rake:28:in rescue in block in file_fetch' /home/logstash/rakelib/fetch.rake:22:inblock in file_fetch'
/home/logstash/rakelib/fetch.rake:30:in file_fetch' /home/logstash/rakelib/vendor.rake:77:inblock (2 levels) in <top (required)>'
/home/logstash/rakelib/z_rubycheck.rake:28:in <top (required)>' /usr/local/share/gems/gems/rake-12.0.0/exe/rake:27:in<top (required)>'

pkempia · May 17, 2017, 10:00am

An update to the above posting.
I followed the steps described in the https://github.com/elastic/logstash/issues/6044

And did modify a file ./logstash/rakelib/fetch.rake to have the proxy variables
Was able to proceed further.

[root@pk-logstash logstash]# rake bootsrap
Downloading http://jruby.org.s3.amazonaws.com/downloads/1.7.25/jruby-bin-1.7.25.tar.gz
Installing minitar 0.5.4 because the build process needs it.
mkdir -p build
mkdir -p build/bootstrap
[bootstrap] Fetching and installing gem: minitar (0.5.4)
Fetching: minitar-0.5.4.gem (100%)
Successfully installed minitar-0.5.4
1 gem installed

But I am again blocked at the similar kind of error found in the logstashissue#6044
i.e
May 17, 2017 9:49:05 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 9:49:05 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
May 17, 2017 9:49:06 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 9:49:06 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
May 17, 2017 9:49:21 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 9:49:21 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
Too many retries, aborting, caused by Bundler::InstallError
rake aborted!

I have tried exporting JRUBY_OPTS with the proxy settings.
Tried adding a new settings.xml and changed ./vendor/bundle/jruby/1.9/gems/ruby-maven-libs-3.3.9/maven-home/conf/settings.xml for the proxies.

Has one any build logstash from source behind a proxy? We plan build a logstash docker.

pjanzen · May 17, 2017, 11:03am

You can create a settings.xml in the users home dir.

Try this:
mkdir ~/.m2
vi ~/.m2/settings.xml

paste the following in the file

<settings>
  <proxies>
   <proxy>
      <id>example-proxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>1.2.3.4</host>
      <port>8080</port>
    </proxy>
  </proxies>
</settings>

pkempia · May 17, 2017, 11:14am

Thanks pjanzen.

did create the file.

[root@pk-logstash logstash]# cat ~/.m2/setttings.xml
<settings>
  <proxies>
   <proxy>
      <id>proxy</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>www-proxy.xyz.com</host>
      <port>80</port>
    </proxy>
  </proxies>
</settings>

Still no luck.

[root@pk-logstash logstash]# rake bootstrap
Invoking bundler install...
May 17, 2017 11:10:52 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 11:10:52 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443

pjanzen · May 17, 2017, 11:27am

Ok, for logstash-plugin I use the following setting as well to get things working.

export JRUBY_OPTS="$JRUBY_OPTS -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-noverify -X-C -Xcompile.invokedynamic=false -J-Dhttp.proxyHost=proxy-server.com -J-Dhttp.proxyPort=443"

I have edit the /usr/share/logstash/bin/logstash-plugin file for that so you have to look which file you need to adjust. (mabey just a export will do, I dont know.)

pkempia · May 17, 2017, 11:36am

No much luck, pjanzen. still fails with the similar error.

[root@pk-logstash logstash]# export JRUBY_OPTS="$JRUBY_OPTS -J-XX:+TieredCompilation -J-XX:TieredStopAtLevel=1 -J-noverify -X-C -Xcompile.invokedynamic=false -J-Dhttp.proxyHost=http://www-proxy.us.xyz.com -J-Dhttp.proxyPort=80"
[root@pk-logstash logstash]#
[root@pk-logstash logstash]#
[root@pk-logstash logstash]#
[root@pk-logstash logstash]# rake bootstrap
Invoking bundler install...
May 17, 2017 11:31:46 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 11:31:46 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443
May 17, 2017 11:31:53 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.NoRouteToHostException) caught when processing request to {s}->https://repo.maven.apache.org:443: No route to host (Host unreachable)
May 17, 2017 11:31:53 AM org.apache.maven.wagon.providers.http.httpclient.impl.execchain.RetryExec execute
INFO: Retrying request to {s}->https://repo.maven.apache.org:443

You seem to mention about this file :/usr/share/logstash/bin/logstash-plugin. is this attached?

pjanzen · May 17, 2017, 11:56am

No the file is not attached, i merly stated this as how I once fixed the problem.

So exporting the http_proxy and https_proxy and setting.xml and export JRUBY_OPTS does not work?

You are sure the proxy is allowed access to the outside world?

...
looking closer at the url rake is using, you should do export https_proxy= I think that will work..

pjanzen · May 17, 2017, 12:00pm

You can also try and set the proxy in ~/.gemrc more on the topic here

pkempia · May 17, 2017, 12:05pm

Yes, none of the settings "http_proxy and https_proxy and setting.xml and export JRUBY_OPTS" seem to work.
I do understand, yes, my proxy allows me the access the external world.

> i[root@pk-logstash logstash]# curl www.google.co.in

<!doctype html>Google(function(){window.google={kEI:'jTscWc72E8qGjwTT6rTYCg',kEXPI:'201761,1352552,3700269,3700347,3700410,3700425,4029815,4031109,4032677,4036527,4039268,4043492,4045841,4048347,4065786,4072126,4072364,4072773,4076096,4076999,4078430,4079444,4081038,4081165,4082164,4084977,4090550,4090554,4092628,4094039,4094251,4094544,4095910,4096323,4097153,4097922,4097929,4097951,4098096,4098721,4098728,4098752,4100169,4100174,4100689,4100717,4101429,4101430,4101437,4101750,4102239,4102974,4103254,4103475,4103845,4104204,4104620,4105085,4105178,4105264,4105319,4105323,4105553,4105732,4106949,4107395,4107417,4107555,4107628,4108495,4108538,4108540,4109316,4109438,4109490,4109498,4109524,4109631,4110233,4110361,4110656,4111030,4111423,4111609,4111792,4112041,4112152,8503585,8508229,8508931,8509037,8509091,8509373,8510344,10200083,10201956,10202317,16200026,19002115,19002174,19002281,19002294,19002296,19002297,41027342',authuser:0,kscs:'c9c918f0_24'};google.kHL='en-IN';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};

pkempia · May 17, 2017, 12:16pm

No pjazen, still no luck. Created that .gemrc file as suggested. quite desperate to get this thing going
. And thanks for support and the suggestions. Is there a way, to figure out where in bundler the call is made in the code?
I had a change done in ./logstash/rakelib/fetch.rake to include the proxy settings and that seemed to work to fetch the jruby, which was failing earlier.

pjanzen · May 17, 2017, 12:19pm

I am at a lose as well, my ruby knowledge is lacking to say the least. A round on google tells me there is a rake-proxy package, maybe that will help?

I am out of options...

system · June 14, 2017, 12:19pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.