# Update scripting to automate REST call to external analytic service? **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2013, 5:36pm UTC](https://discuss.elastic.co/t/update-scripting-to-automate-rest-call-to-external-analytic-service/14793 "2013-12-10T17:36:29Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Josh\_Harrison](https://avatars.discourse-cdn.com/v4/letter/j/0ea827/32.png) [@Josh\_Harrison](https://discuss.elastic.co/u/Josh_Harrison)\ **Post date:** [December 10, 2013, 5:36pm UTC](https://discuss.elastic.co/t/update-scripting-to-automate-rest-call-to-external-analytic-service/14793/1 "2013-12-10T17:36:29Z")
Say I have a tool that lets me take a shortened URL like {"url":"[http://goo.gl/TTX8j5](http://goo.gl/TTX8j5)"} and return the expanded URL like {"url": "[http://goo.gl/TTX8j5](http://goo.gl/TTX8j5)", "resolved\_url": "[http://www.elasticsearch.org/](http://www.elasticsearch.org/)"} It will even do this when I route through, say, goo.gl to bit.ly to tinyurl, and then finally to the final page. the "expanded\_url" field present in the twitter stream only seems to go one hop. Say I have a couple hundred million records already loaded that I'd like to go back and expand the URLs for. I was all set to go start pulling records out with Python (my preferred language), updating them, and bulk loading them back in. But then I realized that you can script with the bulk update function. Can I, with a data structure like follows, send every entities.urls.url to my rest call at localhost:8888/urlexpansion, and insert the results for resolved\_url into the entities.urls structure with the update scripting function? Further, is there a practical way for me to extract all of the IDs of all of the records that have objects in the entities.urls object? "\_source": { "filter\_level": "medium", "contributors": null, "text": "", "geo": null, "retweeted": false, "in\_reply\_to\_screen\_name": null, "truncated": false, "lang": "und", "entities": { "symbols": [], ``` "urls": [{ "url": "https://t.co/XdXRudPXH5", "expanded_url": "https://blog.twitter.com/2013/rich-photo-experience-now-in-embedded-tweets-3", "display_url": "blog.twitter.com/2013/rich-phot\u2026", "indices": [80, 103] }] "hashtags": [], "user_mentions": [] }, "in_reply_to_status_id_str": null, "id":, "source": "Twitter for iPhone", "in\_reply\_to\_user\_id\_str": null, "favorited": false, "in\_reply\_to\_status\_id": null, "retweet\_count": 0, "created\_at": "Tue Nov 26 01:03:06 +0000 2013", "in\_reply\_to\_user\_id": null, "favorite\_count": 0, "id\_str": "123", "place": null, "user": { ... }, "coordinates": null } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cf62a331-4e50-46df-a5cd-befff58fbe19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf62a331-4e50-46df-a5cd-befff58fbe19%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 11, 2013, 8:22am UTC](https://discuss.elastic.co/t/update-scripting-to-automate-rest-call-to-external-analytic-service/14793/2 "2013-12-11T08:22:55Z")
Hey, just check [https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/update/UpdateByNativeScriptTests.java](https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/update/UpdateByNativeScriptTests.java) you can simply add the update request to a bulk request and you are done... and have a native script doing the HTTP lookups. But to be honest, I would not do this inside of elasticsearch, but rather execute the lookup via an external system (or python script) and then execute the update API and update only that field which shows the URL. (As this is a blocking call and needs to wait for a HTTP request to return, I would not delegate that to elasticsearch, as it is a search engine). Hope this helps... --Alex On Tue, Dec 10, 2013 at 6:36 PM, Josh Harrison [hijakk@gmail.com](mailto:hijakk@gmail.com) wrote: > Say I have a tool that lets me take a shortened URL like {"url":" > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://goo.gl/TTX8j5)"} and return the expanded URL like {"url": " > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://goo.gl/TTX8j5)", "resolved\_url": "[http://www.elasticsearch.org/](http://www.elasticsearch.org/)"} > It will even do this when I route through, say, goo.gl to bit.ly to > tinyurl, and then finally to the final page. the "expanded\_url" field > present in the twitter stream only seems to go one hop. > Say I have a couple hundred million records already loaded that I'd like > to go back and expand the URLs for. I was all set to go start pulling > records out with Python (my preferred language), updating them, and bulk > loading them back in. But then I realized that you can script with the bulk > update function. > Can I, with a data structure like follows, send every entities.urls.url to > my rest call at localhost:8888/urlexpansion, and insert the results for > resolved\_url into the entities.urls structure with the update scripting > function? > Further, is there a practical way for me to extract all of the IDs of all > of the records that have objects in the entities.urls object? > > "\_source": { > "filter\_level": "medium", > "contributors": null, > "text": "", > "geo": null, > "retweeted": false, > "in\_reply\_to\_screen\_name": null, > "truncated": false, > "lang": "und", > "entities": { > "symbols": , > > ``` > "urls": [{ > "url": "https://t.co/XdXRudPXH5", > "expanded_url": "https://blog.twitter.com/2013/rich-photo-experience-now-in-embedded-tweets-3", > "display_url": "blog.twitter.com/2013/rich-phot\u2026 ", > "indices": [80, 103] > }] > > "hashtags": [], > "user_mentions": [] > }, > "in_reply_to_status_id_str": null, > "id":, > "source": " > ``` > > rel="nofollow"\>Twitter for iPhone", > "in\_reply\_to\_user\_id\_str": null, > "favorited": false, > "in\_reply\_to\_status\_id": null, > "retweet\_count": 0, > "created\_at": "Tue Nov 26 01:03:06 +0000 2013", > "in\_reply\_to\_user\_id": null, > "favorite\_count": 0, > "id\_str": "123", > "place": null, > "user": { > ... > }, > "coordinates": null > } > } > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/cf62a331-4e50-46df-a5cd-befff58fbe19%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cf62a331-4e50-46df-a5cd-befff58fbe19%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9rM5%3D%2Bkp%2BYZjj8f6BVWEy3zkEjPMD\_BS0HaYoYzmv9sg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9rM5%3D%2Bkp%2BYZjj8f6BVWEy3zkEjPMD_BS0HaYoYzmv9sg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/update-scripting-to-automate-rest-call-to-external-analytic-service/14793/3 "2017-07-06T02:02:02Z")