Shipping logs via Proxy with Logstash

Elastic Stack Logstash
1

Hi,

How do I tell Logstash to ship logs via a proxy?
I searched and searched, but could not find any documentation about shipping via a proxy. Maybe sending data through proxy is simply not supported at this time?

Thanks,
Otis

2

Which output plugin are you using to ship the logs?

3

Hi Magnus,

Trying to use the Elasticsearch output.
I don't think that output itself has any support for sending data through a proxy, or does it?

I was thinking Logstash must be able to get data out via an HTTP proxy in general.... but maybe it simply doesn't have tht functionality?

Thanks,
Otis

4

I have no idea if it's supported. I'd be surprised if the underlying HTTP library didn't support it so it's quite possibly just a matter of exposing the configuration.

5

I am having the same problem. I have even added "set -x" in "./bin/logstash.sh.lib" to see what gets passed to Ruby. It looks fine to me (servernames and tokens replaced for confidentiality-reasons):

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ export LS_JAVA_OPTS="-Dhttp.proxyHost=proxy-srv.mycorp.net  -Dhttp.proxyPort=8080"
HSBX01@my-server /opt/oracle/logstash-1.5.1 $ ./bin/logstash -f hsbx01-sematext.conf            
..
+ JAVA_OPTS='-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Dhttp.proxyHost=proxy-srv.mycorp.net  -Dhttp.proxyPort=8080 -Xmx500m'
..
+ ruby_exec /opt/oracle/logstash-1.5.1/lib/bootstrap/environment.rb logstash/runner.rb agent -f hsbx01-sematext.conf
..
++ for i in '$JAVA_OPTS'
++ printf %s ' -J-Dhttp.proxyHost=proxy-srv.mycorp.net'
++ for i in '$JAVA_OPTS'
++ printf %s ' -J-Dhttp.proxyPort=8080'
++ for i in '$JAVA_OPTS'
++ printf %s ' -J-Xmx500m'
+ exec /opt/oracle/logstash-1.5.1/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J-Dhttp.proxyHost=proxy-srv.mycorp.net -J-Dhttp.proxyPort=8080 -J-Xmx500m /opt/oracle/logstash-1.5.1/lib/bootstrap/environment.rb logstash/runner.rb agent -f hsbx01-sematext.conf
Logstash startup completed
my Logstash-Test
Jul 08, 2015 5:42:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}->http://logsene-receiver.sematext.com:80: Connect to logsene-receiver.sematext.com:80 [logsene-receiver.sematext.com/23.21.237.54, logsene-receiver.sematext.com/23.21.74.13] failed: Connection refused

Here is my Logstash-configuration:

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat hsbx01-sematext.conf
input {
  stdin { }
}

output {
  elasticsearch {
    protocol => "http" # you should not change this to https if you want to use SSL
    # ssl => "false" # if you want to use SSL/HTTPS
    host => "logsene-receiver.sematext.com"
    port => 80 # use 443 if you enable SSL
    index => "<stripped>" # your token
    manage_template => false
  }
}

However, I also tried a test script from some forum to test Ruby itself. Apparently, Ruby is able to communicate with outside world and even with the particular server that is configured in the Logstash-configuration:

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat conn-test.sh 
#!/bin/bash 
export JAVA_OPTS="-Dhttp.proxyHost=proxy-srv.mycorp.net -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*.dont.proxy.me" 
vendor/jruby/bin/jruby ./conn-test.rb 
HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat conn-test.rb 
require 'net/http' 
require 'java' 
java_import  java::lang::System 
uri=URI('http://logsene-receiver.sematext.com') 
puts "Proxy #{System.getProperty('http.proxyHost')}:#{System.getProperty('http.proxyPort')}" 
puts "Proxy exclusion: #{System.getProperty('http.nonProxyHosts')}" 
puts  Net::HTTP.get(uri) 

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ ./conn-test.sh 
Proxy proxy-srv.mycorp.net:8080 
Proxy exclusion: *.dont.proxy.me 
{"status":200,"name":"Sematext-Logsene","cluster_name":"elasticsearch","version":{"number":"1.4.4","build_hash":"c88f77ffc81301dfa9dfd81ca2232f09588bd512","build_timestamp":"2015-02-19T13:05:36Z","build_snapshot":false,"lucene_version":"4.10.3"},"tagline":"You Know, for Search"} 
HSBX01@my-server /opt/oracle/logstash-1.5.1 $

So I think I am missing something obvious, I just don't know yet what... Does anyone have an idea?

6

We added support for this in LS 1.5.3. For details see: https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/199

Docs here: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-proxy

7

The new 1.5.3 proxy output feature seems to only apply to ElasticSearch. Is there a way to send http output to a specified proxy using only Logstash, with no ElasticSearch component?

According to a old StackOverflow post this kind of thing should work, but it doesn't:
export LS_JAVA_OPTS="-Dhttp.proxyHost=proxy.somesite.com -Dhttp.proxyPort=1234 -Dhttps.proxyHost=proxy.somesite.com -Dhttps.proxyPort=1234"

8

In 5.0.0 ALPHA1, the ability to ship from Beats to Logstash via a SOCKS5 proxy was added as well.

9

I just have face this problem, that i use Logstash to ship logs to webhdfs through a proxy. I just use "export LS_JAVA_OPTS="-Dhttp.proxyHost=ur-proxy -Dhttp.proxyPort=ur-port", it work fine for logstash version 5.1