Best practices for sending Logstash output duplicates off-site


#1

Hi,

We are getting Beats events and Luberjack input to our logstash.
We then do some filtering and store these into our Elasticsearch.
However, we also need to send this output off-site through a HTTP proxy.
We have a configuration that works fine, provided that there are no issues with the off-site connection. Alas, once the off-site connection has a hiccup, it obviously stalls also the output to our own local Elasticsearch.

What would be the best practice approach here? I understand from my research that two pipelines would keep the outputs from interfering with one another? However, the pipelines could not listen to the same ports, so the traffic would need duplicating at the sending Beats, or maybe with logstash (from one pipeline to two others)? Would backpressure still influence both outputs?

Here is our config. Thanks for the input :slight_smile:

input {
    beats {
        port => 7000
    }
    lumberjack {
        port => 7001
        id => "xxx_Lumberjack"
        ssl_certificate => "../logstash-forwarder.crt"
        ssl_key => "../logstash-forwarder.key"
        codec => json
    }
}
filter {
    ...
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
        manage_template => false
        index => "%{els_index}-%{+YYYY.MM.dd}"
    }
    http {
        proxy => {
            host => "xxx.xxx.xxx.xxx"
            port => 8080
            scheme => "http"
            user => "xxxxxx"
            password => "xxxxxx"
        }
        http_method => put
        cacert => "../cert/server.pem"
        format => "json"
        content_type => "application/json;charset=UTF-8"
        url => "https://xxxxxx"
        headers => ["Authorization", "Basic xxxxx"]
    }
}

(Magnus B├Ąck) #2

Feed your inbound events into a message broker with two queues, one for the offsite stuff and one for the rest. Let the broker queue up the messages when there's a hiccup.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.