we are trying to enrich out firewall/router logs with custom geoip data (such as coordinates, client info, internal client ID,...), which we have saved in a dedicated index on our ES cluster.
I'm currently trying out solution with logstash-filter-elasticsearch plugin, but the performance is really poor: cca 9k event/s without the filter and only 1200k events/s with the filter enabled. And since we want to enrich src and dst IPs, that comes down to cca 700k events/s.
Looking at netstat, it looks like the filter is opening a new connection for every query. Adding the latency (since ES and logstash are not on the same servers), it comes down to poor performance. Is there a way for this plugin to keep connection opened?
What can I do to get better performance?
Any other idea how to achieve the same result? I would like to avoid building my own geoip (maxmind like) binary database if possible.