# Logstash slows down overtime

**URL:** <https://discuss.elastic.co/t/logstash-slows-down-overtime/184274>\
**Category:** Logstash\
**Created:** [June 5, 2019, 4:31am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274 "2019-06-05T04:31:49Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 4:31am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/1 "2019-06-05T04:31:49Z")

</div>

> input {  
> file {  
> path =\> "/home/elk/logstash-6.5.4/input\_data/log-1000k.log"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> }  
> }  
> filter {  
> kv {  
> field\_split =\> " "  
> value\_split =\> "="  
> }
> 
> ruby {  
> init =\> "  
> require 'net/http'  
> require 'json'  
> require 'uri'  
> "  
> code =\> "
> 
> ```
> uri = URI.parse('https://xxxxxxxxxx/xxxxxx/servlet/xxxxxxx')
> #check if field exist
> if event.get('srcip')
> srcip = event.get('srcip')
> else	
> srcip = ''
> end	
> if event.get('poluuid')
> poluuid = event.get('poluuid')
> else	
> poluuid = ''
> end	
> #add params				
> params = { :hash_pair => ['srcip' + '-' + srcip, 'poluuid' + '-' + poluuid] }
> uri.query = URI.encode_www_form( params )
>   
> response = Net::HTTP.get_response(uri)
> if response.code == '200'
> 
> result = JSON.parse(response.body)
> #replace fields in order of hash values
> if event.get('srcip')
> event.set('[srcip]', result['srcip'])
> end	
> if event.get('poluuid')
> event.set('[poluuid]', result['poluuid'])
> end	
> 
> else
> event.set('echo','ERROR reaching web service')
> end
> "
> 
> ```
> 
> }
> 
> mutate { remove\_field =\> ["path"] }  
> }  
> output {  
> elasticsearch {  
> }  
> }

Dear All,

I am using this config, which part will cause the slow down over time for logstash?  
I keep refreshing Kibana \_count for index,  
It is fast at the beginning but it slows down overtime, please help! Thanks!

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 4:52am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/2 "2019-06-05T04:52:01Z")

</div>

I discovered that the CPU usage is increasing overtime!  
What is the solution for this?

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 7:19am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/3 "2019-06-05T07:19:39Z")

</div>

For more information, I find that even I do not use any filter but just output to elasticsearch,  
The CPU usage is in the same performance which is keep increasing to the max and the output eventually stopped.  
I am trying with 1m data and CPU reach max at around 200k data,  
Please help urgent!! Thank you.

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [June 5, 2019, 7:54am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/4 "2019-06-05T07:54:28Z")

</div>

What is the machine you are running it on?  
How much RAM? What is the CPU?  
Why do you use the ruby to call external service for status in that fashion for getting the values? It looks slow as you waste timie for every http call you do.  
Does it run Kibana and ES as well or its a separated system?

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 8:00am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/5 "2019-06-05T08:00:25Z")

</div>

it is running on linux, 16gb ram and 16core,  
Logstash, ES and Kibana are on 3 separated server

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 8:04am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/6 "2019-06-05T08:04:16Z")

</div>

I think the ruby call is not the problem causing my problem,  
because I have tried to remove the ruby filter and run again,  
the same problem happened

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [June 5, 2019, 8:07am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/7 "2019-06-05T08:07:13Z")

</div>

What do you use in your jvm.options for logstash under -Xms and -Xmx?

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 8:11am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/8 "2019-06-05T08:11:23Z")

</div>

-Xms1g  
-Xmx1g

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [June 5, 2019, 8:14am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/9 "2019-06-05T08:14:47Z")

</div>

Could you increase that value to 4g and run your test again?  
You will need to reload your logstash.

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 8:16am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/10 "2019-06-05T08:16:45Z")

</div>

how is this related to increasing CPU usage?  
Thanks

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [June 5, 2019, 8:25am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/11 "2019-06-05T08:25:02Z")

</div>

Maybe the garbage collector is busy and you get that CPU load because of that. Its just a guess because GCs are doing pauses between reducing the memory footprint.

If you want to do proper debugging on this, you need to start profiling your application and check where and what consumes that power. You have a few JVM profiles out there.

Increasing the RAM is for the lazy ones that does. Does not hurt to check if your problem will go away.

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 5, 2019, 9:06am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/12 "2019-06-05T09:06:41Z")

</div>

Sorry for a little bit off topic,  
Is that GC works like if JVM reach 1g, GC will start working until memories are freed and repeat?  
If yes, I think your point make sense. Thanks.

---

<div class="post-metadata">

**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\
**Post date:** [June 6, 2019, 3:23am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/13 "2019-06-06T03:23:35Z")

</div>

@pastechecker  
I have tried to increase the value to 4g and seems it have improved  
before increase the value, I can output about 230k data to elasticsearch, After changed, I can output 290k, and stopped since the cpu usage reached the max.  
Is that my hardware is not capable to output 1000k data?

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [June 6, 2019, 9:43am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/14 "2019-06-06T09:43:04Z")

</div>

What does the logs say? You could have a look for ERRORS and FATALS. Is your elasticsearch also configured to 1GB of heap size?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 6, 2019, 10:21am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/15 "2019-06-06T10:21:22Z")

</div>

Logstash can only processdata as fast as Elasticsearch can accept it. What is the specification of your Elasticsearch cluster? Is there anything in the Elasticsearch logs that indicate a problem?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 4, 2019, 10:21am UTC](https://discuss.elastic.co/t/logstash-slows-down-overtime/184274/16 "2019-07-04T10:21:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.