# New Plugin: logstash-filter-augment - Enhance your events from external data files **URL:** \ **Category:** Logstash\ **Created:** [January 18, 2017, 5:47pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062 "2017-01-18T17:47:17Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Adam\_Caldwell](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Caldwell](https://discuss.elastic.co/u/Adam_Caldwell)\ **Post date:** [January 18, 2017, 5:47pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/1 "2017-01-18T17:47:17Z")
I wrote a plugin that people here might find useful. Given a CSV/JSON/YML file with structured data, add fields to your event based on a lookup into that file. The use-case I had for it was to do geocoding based on account number. I created a CSV file that had accountnumber,lat,lon as fields and then used the plugin to lookup the account number and add a [location] element to my events. The filter looks at the file modified time and merges in any changes when the file changes, so it can be use to decouple a database lookup -- ie extract your data from the database and then put the file on the logstash server. If anyone is interested, it's on [rubgems.org](http://rubgems.org): [https://rubygems.org/gems/logstash-filter-augment](https://rubygems.org/gems/logstash-filter-augment) and github: [https://github.com/alcanzar/logstash-filter-augment/](https://github.com/alcanzar/logstash-filter-augment/) I'd welcome any bug-reports/feature requests. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2017, 1:35pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/2 "2017-01-22T13:35:40Z")
How is this different filter different from the translate filter? ---
**Author:** ![Adam\_Caldwell](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Caldwell](https://discuss.elastic.co/u/Adam_Caldwell)\ **Post date:** [January 22, 2017, 3:35pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/3 "2017-01-22T15:35:44Z")
Translate can only add 1 field to an event. Augment can add multiple fields at once. The use case I wrote it for was to add lat/lon based on an account number. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2017, 3:36pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/4 "2017-01-22T15:36:41Z")
Okay. You could add multiple fields with a translate + json combo, though. ---
**Author:** ![Adam\_Caldwell](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Caldwell](https://discuss.elastic.co/u/Adam_Caldwell)\ **Post date:** [January 22, 2017, 4:33pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/5 "2017-01-22T16:33:08Z")
the other use case I see is combining data from an external source. For example you have a username in your logs, but you want to add details on their name, business unit, groups, etc. You can periodically extract a full dump from your source of truth to a CSV file and then augment your events with that information. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2017, 6:22pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/6 "2017-01-22T18:22:29Z")
The translate filter works fine for that use case. ---
**Author:** ![Adam\_Caldwell](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Caldwell](https://discuss.elastic.co/u/Adam_Caldwell)\ **Post date:** [January 22, 2017, 7:06pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/7 "2017-01-22T19:06:28Z")
I don't see how. For a csv file it uses the first field as the dictionary key and the second field as the value. You'd have to have a csv file for each attribute you wanted to augment onto the event. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 22, 2017, 8:33pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/8 "2017-01-22T20:33:32Z")
Oh, you gave another example of where one would want to augment an event with multiple fields. That should still be possible with a translate + json combo. ---
**Author:** ![RogerGMartins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rogergmartins/32/15240_2.png) [@RogerGMartins](https://discuss.elastic.co/u/RogerGMartins)\ **Post date:** [February 6, 2017, 3:51pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/9 "2017-02-06T15:51:52Z")
Hi, i'm a bit stumped, i want to implement this use case and i can not figure out how to handle lon/lat with translate+json combo. I dont have much experience with ELK. If you have some time, would you be able to post 1 example of this configuration? Thanks in advance! (ps:Decided to ask here 'cos google got me here while searching for the subject, i bet others ended up in this thread as well) ---
**Author:** ![Adam\_Caldwell](https://avatars.discourse-cdn.com/v4/letter/a/bbe5ce/32.png) [@Adam\_Caldwell](https://discuss.elastic.co/u/Adam_Caldwell)\ **Post date:** [February 6, 2017, 4:17pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/10 "2017-02-06T16:17:16Z")
With augment, your .json file would look like ``` { "1001": { "lat":123,"lon":23 }, "1002": {"lat":43,"lon":43 } } ``` The logstash config would be: ``` augment { target => "[location]" field => "your_key_field" dictionary_path => "geocode.json" } ``` I'd assume it'd be similar with translate. You would also have to add a mapping to your index to make `location` a `geo_point` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 6, 2017, 4:17pm UTC](https://discuss.elastic.co/t/new-plugin-logstash-filter-augment-enhance-your-events-from-external-data-files/72062/11 "2017-03-06T16:17:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.