# Logstash \<- Kafka over WAN

**URL:** <https://discuss.elastic.co/t/logstash-kafka-over-wan/78251>\
**Category:** Logstash\
**Created:** [March 11, 2017, 7:35pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251 "2017-03-11T19:35:08Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![shaharmor](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@shaharmor](https://discuss.elastic.co/u/shaharmor)\
**Post date:** [March 11, 2017, 7:35pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251/1 "2017-03-11T19:35:08Z")

</div>

Hi,

I'm looking for the best configuration for a high load system where Logstash is in the Azure cloud, and Kafka is in AWS.

At the moment I have 3 Kafka servers with 20 partitions for the topic.

I've created an auto scaling group in Azure for all the logstash servers, but even than I can't get them to reach full usage of the CPU (They have 4 CPU's each), which means I'm waisting resources, and the queue keeps filling up.

Any ideas how I can optimize the configuration for over wan pulling?  
thanks.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 12, 2017, 1:43pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251/2 "2017-03-12T13:43:50Z")

</div>

What does your current Logstash configuration look like? Providing this will probably make it easier for the community to help.

---

<div class="post-metadata">

**Author:** ![shaharmor](https://avatars.discourse-cdn.com/v4/letter/s/58f4c7/32.png) [@shaharmor](https://discuss.elastic.co/u/shaharmor)\
**Post date:** [March 12, 2017, 8:23pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251/3 "2017-03-12T20:23:24Z")

</div>

Right now I'm using this:

```auto
input {
    kafka {
        group_id => "azure"
        rebalance_max_retries => 100
        consumer_threads => 1
        topic_id => "my-topic"
        fetch_message_max_bytes => 5242880
        zk_connect => "zk1:2181,zk2:2181,zk3:2181"
    }
}

filter {
    date {
        match => ["timestamp", "UNIX_MS"]
        remove_field => ["timestamp"]
    }
}

output {
    elasticsearch {
        hosts => ["es1", "es2", "es3", "es4"]
        manage_template => false
        workers => 20
        flush_size => 2000
        idle_flush_time => 5
    }
}

```

---

<div class="post-metadata">

**Author:** ![eperry](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eperry/32/551_2.png) [@eperry](https://discuss.elastic.co/u/eperry)\
**Post date:** [March 12, 2017, 10:39pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251/4 "2017-03-12T22:39:15Z")

</div>

Not that this is an exact comparison, but maybe some of the idea's will help

I found that you need to make sure you pull multiple events, and compress the data. Also, I found that the reconnect and polling intervals will really hurt your wan as it is very chatty.

```
output {
   #Kafka is what logstash shippers send to
    kafka {
            #Comma seperated list of Kafka brokers
            bootstrap_servers => "zk:9092,......."
            compression_type => "gzip"
            retry_backoff_ms => 30000
            reconnect_backoff_ms => 30000
            linger_ms => 5000
            # Keep it simple for now we just call the topic logstash
            topic_id => '%{dst_index}'
            #workers => 1
    }

```

}

Hope this gives you some idea's

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2017, 10:39pm UTC](https://discuss.elastic.co/t/logstash-kafka-over-wan/78251/5 "2017-04-09T22:39:20Z")

</div>

Closed on April 9, 2017, 10:39pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.