# Write a plugin to query and aggregate results from multiple shards **URL:** \ **Category:** Elasticsearch\ **Created:** [September 15, 2014, 6:57am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780 "2014-09-15T06:57:19Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ElasticSearch\_Users\_](https://avatars.discourse-cdn.com/v4/letter/e/b4bc9f/32.png) [@ElasticSearch\_Users\_](https://discuss.elastic.co/u/ElasticSearch_Users_)\ **Post date:** [September 15, 2014, 6:57am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780/1 "2014-09-15T06:57:19Z")
Hi, I am looking through the sources, and I am not sure whether this is possible. What I am looking to is the possibility to manipulate the SearchRequest object when it reaches the SearchShards level. Since I need to update the object with some value that is shard specific. For this, I was checking the TransportBroadcastOperationAction which actually allows to hit multiple shards and we can inject a SearchService. However, in the response aggregation, we may have to write our own logic to call SearchPhaseController::merge() or something. Not sure if this will be a problem when the same code in ElasticSearch changes over releases. There are also other classes like SearchServiceTransportAction and we can also probably extend TransportSearchTypeAction like the other QAF, DFS\_QAF, QTF, DFS\_QTF, etc. However, what I want to know is whether this is standard practice and should be done this way? Or is there any other plugin that allows me to do this? Thanks, Sandeep -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 15, 2014, 7:38am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780/2 "2014-09-15T07:38:19Z")
See the TransportSearchAction, in the doExecute() method, the SearchRequest is dispatched to several transport actions of the search types. Assuming you write your own custom action: the shard level request is ShardSearchRequest. It is easier to add information to SearchRequest, pass it down, and extract relevant parts from the SearchRequest later. See ShardSearchRequest constructor for repacking the parameters and delegate it to a shard. Maybe it is possible to add info to extrasource. Jörg On Mon, Sep 15, 2014 at 8:57 AM, 'Sandeep Ramesh Khanzode' via elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote: > Hi, > > I am looking through the sources, and I am not sure whether this is > possible. What I am looking to is the possibility to manipulate the > SearchRequest object when it reaches the SearchShards level. > Since I need to update the object with some value that is shard specific. > > For this, I was checking the TransportBroadcastOperationAction which > actually allows to hit multiple shards and we can inject a SearchService. > However, in the response aggregation, we may have to write our own logic to > call SearchPhaseController::merge() or something. Not sure if this will be > a problem when the same code in Elasticsearch changes over releases. > > There are also other classes like SearchServiceTransportAction and we can > also probably extend TransportSearchTypeAction like the other QAF, DFS\_QAF, > QTF, DFS\_QTF, etc. However, what I want to know is whether this is standard > practice and should be done this way? Or is there any other plugin that > allows me to do this? > > Thanks, > Sandeep > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP\_fZ6i1\_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Sandeep\_Ramesh\_Khanz](https://avatars.discourse-cdn.com/v4/letter/s/4491bb/32.png) [@Sandeep\_Ramesh\_Khanz](https://discuss.elastic.co/u/Sandeep_Ramesh_Khanz)\ **Post date:** [September 16, 2014, 6:25am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780/3 "2014-09-16T06:25:47Z")
Appreciate the response as always. Please bear with my technical understanding of ES 🙂 In the TransportSearchAction, the doExecute() delegates to one of the six different search types. It is inside the execute methods of those individual six actions, that they will look at the shards. Correct me if I am wrong. Even if I modify the SearchRequest in this case, I will not be able to identify the shardId that is operating on this request object before doing the modification. Which class/module should I use to write my custom action? If I write a seventh SearchAction type extending TransportSearchTypeAction, then in the executeQuery()/executeFetch() first/second phases, I will probably get a hold of the shardsearchrequest object. Are you saying that I should modify it there? Please let me know. My concern is that if I write a new searchaction custom plugin like this, I will not be able to replicate ES functionality of the existing six search types and I will also be referring to some Internal classes of ES. Will that be okay? Maybe I am missing something. But I need to modify the request object with the insertion of the shardID identifier when it is known. The reason I need to insert the shardID is so that the Filter Parser Plugin can then take a look at the SearchRequest (or ShardSearchRequest at that time) and do something specific for that shard. Thanks, Sandeep On Mon, Sep 15, 2014 at 1:08 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > See the TransportSearchAction, in the doExecute() method, the > SearchRequest is dispatched to several transport actions of the search > types. > > Assuming you write your own custom action: the shard level request > is ShardSearchRequest. It is easier to add information to SearchRequest, > pass it down, and extract relevant parts from the SearchRequest later. See > ShardSearchRequest constructor for repacking the parameters and delegate it > to a shard. > > Maybe it is possible to add info to extrasource. > > Jörg > > On Mon, Sep 15, 2014 at 8:57 AM, 'Sandeep Ramesh Khanzode' via > elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote: > > > Hi, > > > > I am looking through the sources, and I am not sure whether this is > > possible. What I am looking to is the possibility to manipulate the > > SearchRequest object when it reaches the SearchShards level. > > Since I need to update the object with some value that is shard specific. > > > > For this, I was checking the TransportBroadcastOperationAction which > > actually allows to hit multiple shards and we can inject a SearchService. > > However, in the response aggregation, we may have to write our own logic to > > call SearchPhaseController::merge() or something. Not sure if this will be > > a problem when the same code in Elasticsearch changes over releases. > > > > There are also other classes like SearchServiceTransportAction and we can > > also probably extend TransportSearchTypeAction like the other QAF, DFS\_QAF, > > QTF, DFS\_QTF, etc. However, what I want to know is whether this is standard > > practice and should be done this way? Or is there any other plugin that > > allows me to do this? > > > > Thanks, > > Sandeep > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/BS64wfqrHNM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/BS64wfqrHNM/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP\_fZ6i1\_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP\_fZ6i1\_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-\_mB\_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-_mB_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 16, 2014, 7:40am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780/4 "2014-09-16T07:40:15Z")
If you want to use the filter parser plugin - I think you mean [https://github.com/lmenezes/elasticsearch-terms-fetch-filter-plugin](https://github.com/lmenezes/elasticsearch-terms-fetch-filter-plugin) - then why don't you simply extend the plugin and build a new plugin from that codebase? From what I understand is you somehow want to modify the search action core code but that is not the best method how to extend Elasticsearch on the query side. In a plugin you can add new queries and new query filters very easily, they can be registered at plugin start time, without tampering with the low level core code. Jörg On Tue, Sep 16, 2014 at 8:25 AM, Sandeep Ramesh Khanzode \< [k.sandeep.r@gmail.com](mailto:k.sandeep.r@gmail.com)\> wrote: > Appreciate the response as always. > > Please bear with my technical understanding of ES 🙂 > > In the TransportSearchAction, the doExecute() delegates to one of the six > different search types. It is inside the execute methods of those > individual six actions, that they will look at the shards. Correct me if I > am wrong. Even if I modify the SearchRequest in this case, I will not be > able to identify the shardId that is operating on this request object > before doing the modification. > > Which class/module should I use to write my custom action? If I write a > seventh SearchAction type extending TransportSearchTypeAction, then in the > executeQuery()/executeFetch() first/second phases, I will probably get a > hold of the shardsearchrequest object. Are you saying that I should modify > it there? Please let me know. My concern is that if I write a new > searchaction custom plugin like this, I will not be able to replicate ES > functionality of the existing six search types and I will also be referring > to some Internal classes of ES. Will that be okay? > > Maybe I am missing something. But I need to modify the request object with > the insertion of the shardID identifier when it is known. The reason I need > to insert the shardID is so that the Filter Parser Plugin can then take a > look at the SearchRequest (or ShardSearchRequest at that time) and do > something specific for that shard. > > Thanks, > Sandeep > > On Mon, Sep 15, 2014 at 1:08 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > See the TransportSearchAction, in the doExecute() method, the > > SearchRequest is dispatched to several transport actions of the search > > types. > > > > Assuming you write your own custom action: the shard level request > > is ShardSearchRequest. It is easier to add information to SearchRequest, > > pass it down, and extract relevant parts from the SearchRequest later. See > > ShardSearchRequest constructor for repacking the parameters and delegate it > > to a shard. > > > > Maybe it is possible to add info to extrasource. > > > > Jörg > > > > On Mon, Sep 15, 2014 at 8:57 AM, 'Sandeep Ramesh Khanzode' via > > elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote: > > > > > Hi, > > > > > > I am looking through the sources, and I am not sure whether this is > > > possible. What I am looking to is the possibility to manipulate the > > > SearchRequest object when it reaches the SearchShards level. > > > Since I need to update the object with some value that is shard specific. > > > > > > For this, I was checking the TransportBroadcastOperationAction which > > > actually allows to hit multiple shards and we can inject a SearchService. > > > However, in the response aggregation, we may have to write our own logic to > > > call SearchPhaseController::merge() or something. Not sure if this will be > > > a problem when the same code in Elasticsearch changes over releases. > > > > > > There are also other classes like SearchServiceTransportAction and we > > > can also probably extend TransportSearchTypeAction like the other QAF, > > > DFS\_QAF, QTF, DFS\_QTF, etc. However, what I want to know is whether this is > > > standard practice and should be done this way? Or is there any other plugin > > > that allows me to do this? > > > > > > Thanks, > > > Sandeep > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e1f52da2-bb05-4005-bf88-8031f5440225%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/BS64wfqrHNM/unsubscribe](https://groups.google.com/d/topic/elasticsearch/BS64wfqrHNM/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP\_fZ6i1\_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP\_fZ6i1\_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGMr6K81CSWid1%2BoPP_fZ6i1_4J%2B0uNXHuqjCkVB5MXWw%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-\_mB\_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-_mB_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-\_mB\_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKnM90YKM68sc-_mB_yGhP-93N%2BE0fipfzWyX8jDZwX-phPZsA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFfsvA-FpOUuTGOQLkv\_2PxbYny6iN\_n%2Bob3v79E-%2B\_VQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFfsvA-FpOUuTGOQLkv_2PxbYny6iN_n%2Bob3v79E-%2B_VQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:02am UTC](https://discuss.elastic.co/t/write-a-plugin-to-query-and-aggregate-results-from-multiple-shards/19780/5 "2017-07-06T01:02:03Z")