# Best practices for sending Logstash output duplicates off-site

**URL:** <https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289>\
**Category:** Logstash\
**Created:** [January 19, 2018, 6:44pm UTC](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289 "2018-01-19T18:44:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Aceman87](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Aceman87](https://discuss.elastic.co/u/Aceman87)\
**Post date:** [January 19, 2018, 6:44pm UTC](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289/1 "2018-01-19T18:44:22Z")

</div>

Hi,

We are getting Beats events and Luberjack input to our logstash.  
We then do some filtering and store these into our Elasticsearch.  
However, we also need to send this output off-site through a HTTP proxy.  
We have a configuration that works fine, provided that there are no issues with the off-site connection. Alas, once the off-site connection has a hiccup, it obviously stalls also the output to our own local Elasticsearch.

What would be the best practice approach here? I understand from my research that two pipelines would keep the outputs from interfering with one another? However, the pipelines could not listen to the same ports, so the traffic would need duplicating at the sending Beats, or maybe with logstash (from one pipeline to two others)? Would backpressure still influence both outputs?

Here is our config. Thanks for the input 🙂

```auto
input {
    beats {
        port => 7000
    }
    lumberjack {
        port => 7001
        id => "xxx_Lumberjack"
        ssl_certificate => "../logstash-forwarder.crt"
        ssl_key => "../logstash-forwarder.key"
        codec => json
    }
}
filter {
    ...
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "%{els_index}-%{+YYYY.MM.dd}"
    }
    http {
        proxy => {
            host => "xxx.xxx.xxx.xxx"
            port => 8080
            scheme => "http"
            user => "xxxxxx"
            password => "xxxxxx"
        }
        http_method => put
        cacert => "../cert/server.pem"
        format => "json"
        content_type => "application/json;charset=UTF-8"
        url => "https://xxxxxx"
        headers => ["Authorization", "Basic xxxxx"]
    }
}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 23, 2018, 8:47pm UTC](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289/2 "2018-01-23T20:47:44Z")

</div>

Feed your inbound events into a message broker with two queues, one for the offsite stuff and one for the rest. Let the broker queue up the messages when there's a hiccup.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2018, 8:47pm UTC](https://discuss.elastic.co/t/best-practices-for-sending-logstash-output-duplicates-off-site/116289/3 "2018-02-20T20:47:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.