Shipping logs via Proxy with Logstash

(Otis Gospodnetić) #1


How do I tell Logstash to ship logs via a proxy?
I searched and searched, but could not find any documentation about shipping via a proxy. Maybe sending data through proxy is simply not supported at this time?


(Magnus Bäck) #2

Which output plugin are you using to ship the logs?

(Otis Gospodnetić) #3

Hi Magnus,

Trying to use the Elasticsearch output.
I don't think that output itself has any support for sending data through a proxy, or does it?

I was thinking Logstash must be able to get data out via an HTTP proxy in general.... but maybe it simply doesn't have tht functionality?


(Magnus Bäck) #4

I have no idea if it's supported. I'd be surprised if the underlying HTTP library didn't support it so it's quite possibly just a matter of exposing the configuration.

(Klaus F.) #5

I am having the same problem. I have even added "set -x" in "./bin/" to see what gets passed to Ruby. It looks fine to me (servernames and tokens replaced for confidentiality-reasons):

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ export LS_JAVA_OPTS="  -Dhttp.proxyPort=8080"
HSBX01@my-server /opt/oracle/logstash-1.5.1 $ ./bin/logstash -f hsbx01-sematext.conf            
+ JAVA_OPTS='-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly  -Dhttp.proxyPort=8080 -Xmx500m'
+ ruby_exec /opt/oracle/logstash-1.5.1/lib/bootstrap/environment.rb logstash/runner.rb agent -f hsbx01-sematext.conf
++ for i in '$JAVA_OPTS'
++ printf %s ''
++ for i in '$JAVA_OPTS'
++ printf %s ' -J-Dhttp.proxyPort=8080'
++ for i in '$JAVA_OPTS'
++ printf %s ' -J-Xmx500m'
+ exec /opt/oracle/logstash-1.5.1/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J-Dhttp.proxyPort=8080 -J-Xmx500m /opt/oracle/logstash-1.5.1/lib/bootstrap/environment.rb logstash/runner.rb agent -f hsbx01-sematext.conf
Logstash startup completed
my Logstash-Test
Jul 08, 2015 5:42:29 PM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (org.apache.http.conn.HttpHostConnectException) caught when processing request to {}-> Connect to [,] failed: Connection refused

Here is my Logstash-configuration:

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat hsbx01-sematext.conf
input {
  stdin { }

output {
  elasticsearch {
    protocol => "http" # you should not change this to https if you want to use SSL
    # ssl => "false" # if you want to use SSL/HTTPS
    host => ""
    port => 80 # use 443 if you enable SSL
    index => "<stripped>" # your token
    manage_template => false

However, I also tried a test script from some forum to test Ruby itself. Apparently, Ruby is able to communicate with outside world and even with the particular server that is configured in the Logstash-configuration:

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat 
export JAVA_OPTS=" -Dhttp.proxyPort=8080 -Dhttp.nonProxyHosts=*" 
vendor/jruby/bin/jruby ./conn-test.rb 
HSBX01@my-server /opt/oracle/logstash-1.5.1 $ cat conn-test.rb 
require 'net/http' 
require 'java' 
java_import  java::lang::System 
puts "Proxy #{System.getProperty('http.proxyHost')}:#{System.getProperty('http.proxyPort')}" 
puts "Proxy exclusion: #{System.getProperty('http.nonProxyHosts')}" 
puts  Net::HTTP.get(uri) 

HSBX01@my-server /opt/oracle/logstash-1.5.1 $ ./ 
Proxy exclusion: * 
{"status":200,"name":"Sematext-Logsene","cluster_name":"elasticsearch","version":{"number":"1.4.4","build_hash":"c88f77ffc81301dfa9dfd81ca2232f09588bd512","build_timestamp":"2015-02-19T13:05:36Z","build_snapshot":false,"lucene_version":"4.10.3"},"tagline":"You Know, for Search"} 
HSBX01@my-server /opt/oracle/logstash-1.5.1 $ 

So I think I am missing something obvious, I just don't know yet what... Does anyone have an idea?

(Suyog Rao) #6

We added support for this in LS 1.5.3. For details see:

Docs here:


The new 1.5.3 proxy output feature seems to only apply to ElasticSearch. Is there a way to send http output to a specified proxy using only Logstash, with no ElasticSearch component?

According to a old StackOverflow post this kind of thing should work, but it doesn't:
export LS_JAVA_OPTS=" -Dhttp.proxyPort=1234 -Dhttps.proxyPort=1234"

(Peter Dyson) #8

In 5.0.0 ALPHA1, the ability to ship from Beats to Logstash via a SOCKS5 proxy was added as well.

(Hồ Thúc đồng) #9

I just have face this problem, that i use Logstash to ship logs to webhdfs through a proxy. I just use "export LS_JAVA_OPTS="-Dhttp.proxyHost=ur-proxy -Dhttp.proxyPort=ur-port", it work fine for logstash version 5.1

(system) #10