I am trying to integrate this filter into my pipeline to first set keys in one pipeline (about 4m key => value objects, each about 1kb in size), and then in another step later on get these values in another pipeline to enrich events.
The set speeds are very good, but the get performance slows down my pipeline from around 100.000 events/s to 3.000 events/s.
I tested this by commenting out the logstash-filter-memcached-part from the pipeline, and was able to isolate the memcached-filter "get" as the bottleneck. Logstash pipeline UI viewer marks this filter with orange and shows a processing time of about 10-20ms per event. Which is probably due to TCP overhead. But the plugin can't communicate via UDP.
Is there any way to speed up the "get" inside the pipeline, e.g. by tuning my memcached configuration? I am currently running memcached with -m 5192 as the sole parameter.
I understand the way this filter works (or the nature of the logstash pipeline itself) means that buffered gets, which are much faster than single ones, are out of the question, correct?
Thank you in advance for any pointers.