I use logstash to solve logs, and include some ruby code in logstash's config file. I have total four server to run logstash, and every cpu has 32 cores. I found they can only solve about 10000 logs every second, and the use ratio of the cpu is alreay reach the limitation.
So how to improve the performance of logstash? Is that the ruby code in config file reduce the performance? Any suggest for me?
What does your config look like? What kind of processing are you doing with the ruby filter?
just analyze some field of the log, it looks like below:
indent preformatted text by 4 spaces`if event.include?('decrypted_body')
if event['decrypted_body'] == ''
return
end
if event['body_json'] == nil
event['body_json'] = Hash.new
end
name_hash = Hash['appchannel' => 'app_channel', 'loginchannel' => 'login_channel', 'paychannel' => 'pay_channel']
items = event['decrypted_body'].split('&')
i = 0
while i < items.size do
key_value = items[i].split('=')
i += 1
if key_value.size == 2
if ['appchannel', 'loginchannel', 'paychannel'].include?(key_value[0])
event['body_json'][name_hash[key_value[0]]] = key_value[1]
end
end
end
`