Logstash slows down overtime

Elastic Stack Logstash
1

input {
file {
path => "/home/elk/logstash-6.5.4/input_data/log-1000k.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
kv {
field_split => " "
value_split => "="
}

ruby {
init => "
require 'net/http'
require 'json'
require 'uri'
"
code => "

  		uri = URI.parse('https://xxxxxxxxxx/xxxxxx/servlet/xxxxxxx')
  		#check if field exist
  		if event.get('srcip')
  			srcip = event.get('srcip')
  		else	
  			srcip = ''
  		end	
  		if event.get('poluuid')
  			poluuid = event.get('poluuid')
  		else	
  			poluuid = ''
  		end	
  		#add params				
  		params = { :hash_pair => [ 'srcip' + '-' + srcip, 'poluuid' + '-' + poluuid] }
  		uri.query = URI.encode_www_form( params )
  
  		response = Net::HTTP.get_response(uri)
  		if response.code == '200'

  				result = JSON.parse(response.body)
  				#replace fields in order of hash values
  				if event.get('srcip')
  					event.set('[srcip]', result['srcip'])
  				end	
  				if event.get('poluuid')
  					event.set('[poluuid]', result['poluuid'])
  				end	

  		else
  			event.set('echo','ERROR reaching web service')
  		end
  	"

}

mutate { remove_field => ["path"] }
}
output {
elasticsearch {
}
}

Dear All,

I am using this config, which part will cause the slow down over time for logstash?
I keep refreshing Kibana _count for index,
It is fast at the beginning but it slows down overtime, please help! Thanks!

2

I discovered that the CPU usage is increasing overtime!
What is the solution for this?

3

For more information, I find that even I do not use any filter but just output to elasticsearch,
The CPU usage is in the same performance which is keep increasing to the max and the output eventually stopped.
I am trying with 1m data and CPU reach max at around 200k data,
Please help urgent!! Thank you.

4

What is the machine you are running it on?
How much RAM? What is the CPU?
Why do you use the ruby to call external service for status in that fashion for getting the values? It looks slow as you waste timie for every http call you do.
Does it run Kibana and ES as well or its a separated system?

5

it is running on linux, 16gb ram and 16core,
Logstash, ES and Kibana are on 3 separated server

6

I think the ruby call is not the problem causing my problem,
because I have tried to remove the ruby filter and run again,
the same problem happened

7

What do you use in your jvm.options for logstash under -Xms and -Xmx?

8

-Xms1g
-Xmx1g

9

Could you increase that value to 4g and run your test again?
You will need to reload your logstash.

10

how is this related to increasing CPU usage?
Thanks

11

Maybe the garbage collector is busy and you get that CPU load because of that. Its just a guess because GCs are doing pauses between reducing the memory footprint.

If you want to do proper debugging on this, you need to start profiling your application and check where and what consumes that power. You have a few JVM profiles out there.

Increasing the RAM is for the lazy ones that does. Does not hurt to check if your problem will go away.

12

Sorry for a little bit off topic,
Is that GC works like if JVM reach 1g, GC will start working until memories are freed and repeat?
If yes, I think your point make sense. Thanks.

13

@pastechecker
I have tried to increase the value to 4g and seems it have improved
before increase the value, I can output about 230k data to elasticsearch, After changed, I can output 290k, and stopped since the cpu usage reached the max.
Is that my hardware is not capable to output 1000k data?

14

What does the logs say? You could have a look for ERRORS and FATALS. Is your elasticsearch also configured to 1GB of heap size?

15

Logstash can only processdata as fast as Elasticsearch can accept it. What is the specification of your Elasticsearch cluster? Is there anything in the Elasticsearch logs that indicate a problem?

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.