# How to leverage ES for load balanced processing of data in and ES index **URL:** \ **Category:** Elasticsearch\ **Created:** [March 5, 2012, 7:09am UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886 "2012-03-05T07:09:39Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Mauri](https://avatars.discourse-cdn.com/v4/letter/m/c67d28/32.png) [@Mauri](https://discuss.elastic.co/u/Mauri)\ **Post date:** [March 5, 2012, 7:09am UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886/1 "2012-03-05T07:09:39Z")
I have a situation where I need to perform consolidation, analysis and reporting tasks on data indexed in ES. I would like to leverage ES to load balance or distribute these processing tasks across a group of servers. I am using the java API. I have looked at the possibility of using native scripts and my understanding is that native scripts are executed within the context of queries whereby query requests are sent to all shards in an index and each shard executes the native script locally for each search hit and adds the output of the native script to the search hit. Not quite what I am looking for. The behaviour I am looking for is more like index and get requests which are routed to a shard based on the index routing rules and document key values (index/type/id). Is there a mechanism for executing native scripts for index/put/get requests whereby I could sending a request to ES which gets routed to a cluster node based on a document key and then have that node execute a native script and return a result document generated by that script? The native script may also create new documents which are added to the index for later retrieval. This key based routing would also allow each cluster node to maintain working data in memory or local disc for the tasks it performs. If this is not already supported doe the ES architecture make it possible for me to write a plug in that adds custom actions and action handlers that are compatible with the ES transports and index/get request routing? If this is possible can you direct me to java source code examples of this and/or give me an overview of the ES classes I would have to subclass and how to register these via a plugin. Regards Mauri ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 5, 2012, 3:25pm UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886/2 "2012-03-05T15:25:26Z")
No, there ins't a way to do it. Why not do it outside of ES? On Monday, March 5, 2012 at 9:09 AM, Mauri wrote: > I have a situation where I need to perform consolidation, analysis and > reporting tasks on data indexed in ES. > I would like to leverage ES to load balance or distribute these > processing tasks across a group of servers. > I am using the java API. > > I have looked at the possibility of using native scripts and my > understanding is that native scripts are executed within the context > of queries whereby query requests are sent to all shards in an index > and each shard executes the native script locally for each search hit > and adds the output of the native script to the search hit. Not quite > what I am looking for. > > The behaviour I am looking for is more like index and get requests > which are routed to a shard based on the index routing rules and > document key values (index/type/id). > > Is there a mechanism for executing native scripts for index/put/get > requests whereby I could sending a request to ES which gets routed to > a cluster node based on a document key and then have that node execute > a native script and return a result document generated by that script? > The native script may also create new documents which are added to the > index for later retrieval. This key based routing would also allow > each cluster node to maintain working data in memory or local disc for > the tasks it performs. > > If this is not already supported doe the ES architecture make it > possible for me to write a plug in that adds custom actions and action > handlers that are compatible with the ES transports and index/get > request routing? If this is possible can you direct me to java source > code examples of this and/or give me an overview of the ES classes I > would have to subclass and how to register these via a plugin. > > Regards > Mauri ---
**Author:** ![Mauri](https://avatars.discourse-cdn.com/v4/letter/m/c67d28/32.png) [@Mauri](https://discuss.elastic.co/u/Mauri)\ **Post date:** [March 5, 2012, 11:16pm UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886/3 "2012-03-05T23:16:04Z")
Hi Shay Hoping to leverage ES request routing rather than have to implement something myself or bring in another framework to handle it. Have been looking at the code for the attachment type and it looks feasible to use a custom type as follows: - Create a custom field type 'task' and register a 'TaskMapper' class using a plugin in the same way that AttachmentMapper is registered for the attachment type - Use the index API to index a document containing one or more 'task' fields that contain the parameters for the tasks to be performed. - The index request is routed by ES to a shard based on the key (index/ type/id) assigned to the document - When the document is indexed the parse method of the TaskMapper is called for each task field and TaskMapper processes the task similar to how tika is called by AttachmentMapper Not sure yet how the results generated by the task could be brought out, possibly by passing on to lucene in the way tika output is handled by AttachmentMapper, only to a field with store=true so that the full result is stored in Lucene rather than just being indexed. Regards Mauri ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 6, 2012, 7:26pm UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886/4 "2012-03-06T19:26:50Z")
I thought you said you wanted to generate several documents for a "single json", thats why I said that it makes sense to do it on the client side interacting with elasticsearch. On Tuesday, March 6, 2012 at 1:16 AM, Mauri wrote: > Hi Shay > > Hoping to leverage ES request routing rather than have to implement > something myself or bring in another framework to handle it. > Have been looking at the code for the attachment type and it looks > feasible to use a custom type as follows: > > - Create a custom field type 'task' and register a 'TaskMapper' class > using a plugin in the same way that AttachmentMapper is registered for > the attachment type > - Use the index API to index a document containing one or more 'task' > fields that contain the parameters for the tasks to be performed. > - The index request is routed by ES to a shard based on the key (index/ > type/id) assigned to the document > - When the document is indexed the parse method of the TaskMapper is > called for each task field and TaskMapper processes the task similar > to how tika is called by AttachmentMapper > > Not sure yet how the results generated by the task could be brought > out, possibly by passing on to lucene in the way tika output is > handled by AttachmentMapper, only to a field with store=true so that > the full result is stored in Lucene rather than just being indexed. > > Regards > Mauri ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:37am UTC](https://discuss.elastic.co/t/how-to-leverage-es-for-load-balanced-processing-of-data-in-and-es-index/6886/5 "2017-07-06T03:37:09Z")
Closed on July 6, 2017, 3:37am UTC