# Kafka logstash very high memory use

**URL:** <https://discuss.elastic.co/t/kafka-logstash-very-high-memory-use/49939>\
**Category:** Logstash\
**Created:** [May 12, 2016, 9:11pm UTC](https://discuss.elastic.co/t/kafka-logstash-very-high-memory-use/49939 "2016-05-12T21:11:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![arpan](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@arpan](https://discuss.elastic.co/u/arpan)\
**Post date:** [May 12, 2016, 9:11pm UTC](https://discuss.elastic.co/t/kafka-logstash-very-high-memory-use/49939/1 "2016-05-12T21:11:08Z")

</div>

Hi,

I have logstash 2.3 setup with elasticsearch 2.3. I am using the elasticsearch output and the kafka input. My input consumes from a single topic in kafka and the inputs section looks like this:

input {  
kafka {  
white\_list =\> "topic"  
consumer\_threads =\> 1  
queue\_size =\> 50  
codec =\> plain  
zk\_connect =\> "zk\_path"  
group\_id =\> "logs\_consumers"  
type =\> "log"  
}  
}

output looks like this:  
output {  
if [type] == "log"{  
elasticsearch {  
index =\> "index"  
flush\_size =\> 5000  
idle\_flush\_time =\> 30  
document\_type =\> "logl"  
hosts =\> [hosts]  
}  
}

I first ran it with 4g of ram and the process gave an OutOfMemory Error and died. I tried with memory values all the way up to 7g for the JVM and it still runs out of memory. It seems like this should not be happening.  
I am running with -w 2 and its a 2 core machine. I had a very similar configuration in logstash 1.5 and never ran into this issue.

Any help in figuring out what is causing the massive memory usage here would be very appreciated. Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:57am UTC](https://discuss.elastic.co/t/kafka-logstash-very-high-memory-use/49939/2 "2017-07-06T04:57:48Z")

</div>