# Can I piggy-back on the ec2 discovery mechanism? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 19, 2013, 5:51pm UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029 "2013-05-19T17:51:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 19, 2013, 5:51pm UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/1 "2013-05-19T17:51:43Z")
My application needs to share some state across several ec2 instances (cache, who is master etc). I'm looking at setting up Hazelcast for this, but since each instance of the application is already an elasticsearch node, perhaps there is a way to leverage that instead? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [May 20, 2013, 3:21am UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/2 "2013-05-20T03:21:40Z")
Eric Jain wrote: > My application needs to share some state across several ec2 > instances > (cache, who is master etc). I'm looking at setting up Hazelcast > for this, > but since each instance of the application is already an > elasticsearch > node, perhaps there is a way to leverage that instead? The cache would be tough to piggy-back on. You could grab the master node from cluster state and use it for a custom purpose though. Why not store the data you're looking to share in ES proper? It's already a highly optimized key-value store, among other things. -Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 20, 2013, 6:11am UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/3 "2013-05-20T06:11:01Z")
On Sun, May 19, 2013 at 8:21 PM, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > The cache would be tough to piggy-back on. You could grab the master node > from cluster state and use it for a custom purpose though. Yes, but I just realized that knowing which node is the elasticsearch master doesn't help when what you want is to ensure that a piece of code in your application is run on one machine only. I'd need to do something like "give me the most senior of all nodes tagged with x", which is probably beyond the scope of the elasticsearch discovery mechanism. > Why not store the data you're looking to share in ES proper? It's already a > highly optimized key-value store, among other things. For example, I need to keep track of request counts to enforce quotas; here I wouldn't just have to read, but also update a document for every request! Hazelcast's distributed map works well for this kind of thing, but Hazelcast has its own discovery mechanism that duplicates what elasticsearch is already doing, possibly with its own set of issues. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [May 20, 2013, 3:29pm UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/4 "2013-05-20T15:29:02Z")
Eric Jain wrote: > I'd need to do something like "give me the most senior of all > nodes tagged with x", which is probably beyond the scope of the > elasticsearch discovery mechanism. You can get start\_time from the nodes info API and calculate seniority, if age is what you mean by "seniority." > > Why not store the data you're looking to share in ES proper? > > It's already a highly optimized key-value store, among other > > things. > > For example, I need to keep track of request counts to enforce > quotas; here I wouldn't just have to read, but also update a > document for every request! Hazelcast's distributed map works > well for this kind of thing, but Hazelcast has its own discovery > mechanism that duplicates what elasticsearch is already doing, > possibly with its own set of issues. You might be surprised how efficient ES can be here, especially if you use the update API. How much traffic are you trying to support? Should be fairly easy (he says with no knowledge of your application :)). I'd try it before assimilating a whole new service to maintain. The only stumbling block might be if you have to support such a high volume of data that you have to keep a lot of spare disk capacity on hand, due to the yet-to-be-deleted docs generated from many updates. Since you're looking at an in-memory store in comparison I doubt this is the case, however. Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [May 20, 2013, 5:54pm UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/5 "2013-05-20T17:54:54Z")
On Mon, May 20, 2013 at 8:29 AM, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > You can get start\_time from the nodes info API and calculate seniority, if > age is what you mean by "seniority." That could work; I just need a mechanism to ensure that some code is run on one application instance only. > You might be surprised how efficient ES can be here, especially if you use > the update API. How much traffic are you trying to support? Should be > fairly easy (he says with no knowledge of your application :)). I'd try it > before assimilating a whole new service to maintain. I'll give it a try... Or how about this approach? - for each operation, add a document with two fields ("user", "cost") to a "quotas" index - set the TTL for the document to e.g. 31 days - to check if a user is over quota, use the statistical facet to sum up the "cost" field for a "user" -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [May 21, 2013, 6:36am UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/6 "2013-05-21T06:36:38Z")
Hey, just a quick side note: If you want to make sure, that a piece of code is only once in your cluster, you could simply use a river for that. Elasticsearch takes care of the rest then. If you want to deep dive a bit into elasticsearch and send data yourself to nodes across your cluster, take a look at TransportService.sendRequest(). --Alex On Mon, May 20, 2013 at 7:54 PM, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote: > On Mon, May 20, 2013 at 8:29 AM, Drew Raines [aaraines@gmail.com](mailto:aaraines@gmail.com) wrote: > > > You can get start\_time from the nodes info API and calculate seniority, > > if > > age is what you mean by "seniority." > > That could work; I just need a mechanism to ensure that some code is > run on one application instance only. > > > You might be surprised how efficient ES can be here, especially if you > > use > > the update API. How much traffic are you trying to support? Should be > > fairly easy (he says with no knowledge of your application :)). I'd try > > it > > before assimilating a whole new service to maintain. > > I'll give it a try... > > Or how about this approach? > > - for each operation, add a document with two fields ("user", "cost") > to a "quotas" index > - set the TTL for the document to e.g. 31 days > - to check if a user is over quota, use the statistical facet to sum > up the "cost" field for a "user" > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [May 21, 2013, 1:24pm UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/7 "2013-05-21T13:24:47Z")
Eric Jain wrote: > Or how about this approach? > > - for each operation, add a document with two fields ("user", > "cost") > to a "quotas" index > - set the TTL for the document to e.g. 31 days > - to check if a user is over quota, use the statistical facet to > sum > up the "cost" field for a "user" I'm not quite sure what you're after, but the stats facet is pretty cheap. Sounds good to me. Drew -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:35am UTC](https://discuss.elastic.co/t/can-i-piggy-back-on-the-ec2-discovery-mechanism/12029/8 "2017-07-06T02:35:34Z")