# External lookups in a logstash pipeline **URL:** \ **Category:** Logstash\ **Created:** [June 16, 2017, 6:09pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758 "2017-06-16T18:09:20Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jaspreet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaspreet_singh/32/58296_2.png) [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Post date:** [June 16, 2017, 6:09pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/1 "2017-06-16T18:09:20Z")
So I have a requirement that goes something like this ... 1. An event arrives in kafka 2. Read that event from logstash and get an ID out of it 3. Look up that ID to fetch a JSON from say object storage 4. Do stuff like filtering etc to it 5. Index to Elasticsearch I'm curious about #3 above. What is the best way to do something like that in a logstash pipeline? One possible way out is have a script execute as part of a filter that takes in the ID and dumps the JSON to pipeline after fetching it. But am wondering if logstash supports that. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2017, 9:07pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/2 "2017-06-19T21:07:49Z")
Your best option would probably be a translate or a jdbc\_streaming filter. ---
**Author:** ![Jaspreet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaspreet_singh/32/58296_2.png) [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Post date:** [June 19, 2017, 9:12pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/3 "2017-06-19T21:12:46Z")
thanks for your response @magnusbaeck Can you elaborate a bit more as to how a translate or a jdbc\_streaming filter would help me lookup and fetch a JSON as an event into logstash? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2017, 9:22pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/4 "2017-06-19T21:22:50Z")
Those filters do exactly what you're asking for; they look up a field value in an external data source and stores the result in a field in the current event. Feed the resulting field to a json filter to deserialize the string into fields in the current event. ---
**Author:** ![Jaspreet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaspreet_singh/32/58296_2.png) [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Post date:** [June 21, 2017, 6:03pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/5 "2017-06-21T18:03:45Z")
How about object storage lookup - does jdbc\_streaming handle that? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 22, 2017, 5:14am UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/6 "2017-06-22T05:14:31Z")
I'm not sure exactly what you mean, but any data source that you have a JDBC driver for will work. ---
**Author:** ![Jaspreet\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaspreet_singh/32/58296_2.png) [@Jaspreet\_Singh](https://discuss.elastic.co/u/Jaspreet_Singh)\ **Post date:** [June 22, 2017, 2:44pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/7 "2017-06-22T14:44:02Z")
@magnusbaeck Thanks for your response. So I was referring to any cloud object storage. Let me rephrase this - in case I need to lookup any system that may be doesnt have JDBC driver - im guessing it is going to have to be custom code execution. So im guessing a custom filter or can the ruby filter help too? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 22, 2017, 6:49pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/8 "2017-06-22T18:49:04Z")
I wouldn't use a Ruby filter for anything non-trivial, but theoretically it should work. I'd go with a custom filter. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 20, 2017, 6:49pm UTC](https://discuss.elastic.co/t/external-lookups-in-a-logstash-pipeline/89758/9 "2017-07-20T18:49:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.