# Enriching logs with data from Elasticsearch (logstash-filter-elasticsearch)

**URL:** <https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325>\
**Category:** Logstash\
**Created:** [April 5, 2017, 3:10pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325 "2017-04-05T15:10:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)\
**Post date:** [April 5, 2017, 3:10pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325/1 "2017-04-05T15:10:32Z")

</div>

Hello,

we are trying to enrich out firewall/router logs with custom geoip data (such as coordinates, client info, internal client ID,...), which we have saved in a dedicated index on our ES cluster.

I'm currently trying out solution with logstash-filter-elasticsearch plugin, but the performance is really poor: cca 9k event/s without the filter and only 1200k events/s with the filter enabled. And since we want to enrich src and dst IPs, that comes down to cca 700k events/s.

Looking at netstat, it looks like the filter is opening a new connection for every query. Adding the latency (since ES and logstash are not on the same servers), it comes down to poor performance. Is there a way for this plugin to keep connection opened?

What can I do to get better performance?  
Any other idea how to achieve the same result? I would like to avoid building my own geoip (maxmind like) binary database if possible.

Thanks, Matej

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 5, 2017, 3:19pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325/2 "2017-04-05T15:19:35Z")

</div>

If the data is semi-static, you might be able to do this based on an in-memory translation as described in [this thread](https://discuss.elastic.co/t/create-custom-geoip-database-for-logstash-5-2/79473/6), which should be a lot faster.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [April 5, 2017, 3:35pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325/3 "2017-04-05T15:35:43Z")

</div>

To put some context here, enrichment is a hot topic in Logstash Product management at the moment.  
We have released v1 of a jdbc enrichment filter called `jdbc_streaming`, it has caching so should only have a perf problems on the first occurrence of a new field value. This filter is intended for data that changes fairly regularly e.g. data from a CRM or Transactional DB.  
Next we will release `jdbc_static,` this filter will pull down a record set into local in-memory db and then lookups will be done on the local db using SQL.  
Next we will look at improving the ES filter to add caching.  
Watch the release notes of LS versions to see when this happens or we'll tweet about it.

---

<div class="post-metadata">

**Author:** ![matejzero](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@matejzero](https://discuss.elastic.co/u/matejzero)\
**Post date:** [April 5, 2017, 8:07pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325/4 "2017-04-05T20:07:49Z")

</div>

I though of that, but I don't want to have an entry for every IP address we have. We have probably a few thousand if not more.

Currently, we are saving top and bottom subnet IPs as IP type in ES and then do a search "bottom\<$src\_ip AND top\>$src\_ip". This gives us a single document and we use fields in this document to populate the currently worked-on document.

This probably also wont work for jdbc filter, since I'm not sure sql supports ip type.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2017, 8:07pm UTC](https://discuss.elastic.co/t/enriching-logs-with-data-from-elasticsearch-logstash-filter-elasticsearch/81325/5 "2017-05-03T20:07:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.