A friend and I wrote a codec to parse syslog data for SAR files. The codec works great but performance is slow. What is the best way to profile logstash? VisualVM or jruby profiler? Is there any documentation on this process?
There isn't a pain-free way of doing this right now. We're working to have internal performance metrics in Logstash 2.1 or 2.2, with more features being continuously added thereafter.
Using a jruby profiler works, but it slows things down so dramatically that it almost makes it too slow to tell what's really slowing things down.
You can always to removing bits and pieces of the pipeline and comparing before and after.
If it's definitely in the codec, then that's more of a jruby question. I'd personally start with a reading of the codec code and just looking for any low-hanging fruit.
Thank you for the feedback. I look forward to to the internal performance metrics.
Hi,
You could use SPM today just to get an idea of various JVM metrics for the JVM that runs your Logstash. We've done that here a long time ago: http://blog.sematext.com/2013/11/05/logstash-performance-monitoring/
You'll also get OS metrics, too, which may be relevant here as well. Once Logstash starts exposing metrics you'll see its logo on http://sematext.com/spm/integrations . Looking forward to that!
Otis
Or any of the F/OSS alternatives.
We use https://github.com/jmxtrans/jmxtrans-agent/ combined with
Graphite and are very pleased with the results.