# Logstash crashing Elastic

**URL:** <https://discuss.elastic.co/t/logstash-crashing-elastic/132043>\
**Category:** Logstash\
**Created:** [May 16, 2018, 6:52am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043 "2018-05-16T06:52:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![safhw91](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@safhw91](https://discuss.elastic.co/u/safhw91)\
**Post date:** [May 16, 2018, 6:52am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043/1 "2018-05-16T06:52:50Z")

</div>

I have been parsing logs using logstash's grok parser log by log, and recently one of my pipelines.yml files which contains 100+ input files has been crashing my elastic server (out of memory).

After reading [this](https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html), my guess is Elastic was previously crashing because something wasn't getting cleared out of Logstash's cache while logstash was parsing (~25GB worth of logs) and my system was running out of memory. I have not been running any Kibana queries at all - only parsing with Logstash. Could this possibly be what was making Elastic crash?

Because if not, my only other guess would be that my system simply allocated too little memory to Elastic, which seems like a more expensive problem!

Thank you.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 16, 2018, 6:57am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043/2 "2018-05-16T06:57:48Z")

</div>

How much heap do you have configured for Elasticsearch? How many indices and shards do you have?

---

<div class="post-metadata">

**Author:** ![safhw91](https://avatars.discourse-cdn.com/v4/letter/s/eb9ed0/32.png) [@safhw91](https://discuss.elastic.co/u/safhw91)\
**Post date:** [May 16, 2018, 7:33am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043/3 "2018-05-16T07:33:38Z")

</div>

Thanks for your reply.

At the point of my testing (before I started reading up about memory allocation in Elastic), I was using the default: 1GB heap and only 4 indices. Amount of documents inside before Elastic crashed was about 300 million documents (~40GB).

What I've done after that was to set:

1. indices.fielddata.cache.size: 20%
2. Modify heap size to 8GB

I'm sure one or both of these will keep my server alive for longer, but I just wanted to get to the bottom of what exactly was causing the crash.

P.S. eventually, I'll probably have about 1TB worth of data inside Elastic and somewhere from 8GB-32GB RAM for Elastic.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 13, 2018, 7:33am UTC](https://discuss.elastic.co/t/logstash-crashing-elastic/132043/4 "2018-06-13T07:33:44Z")

</div>

Closed on June 13, 2018, 7:33am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.