Feeds/Shard based on machines hardware capacity

Hi ,

Is there any provision in elasticsearch where it automatically decides on
increasing or decreasing percentage of feed in a particular shard due to
its hardware capability.
I would love to see a feature where it allocates more feeds to a shard
whose machine has bigger capability and lesser feeds whose allocated
machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have
what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks
Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

have you taken a look at shard allocation?
See http://www.elasticsearch.org/guide/reference/index-modules/allocation/

It's not automatic (there is no reliable mechanism deciding for
elasticsearch that this node is extraordinary powerful. Judging from its
hardware might be a mistake as many other services might run on the node,
you have to jump in here) mechanism, but you can do it by grouping nodes
for example.

--Alex

On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

Hi ,

Is there any provision in elasticsearch where it automatically decides on
increasing or decreasing percentage of feed in a particular shard due to
its hardware capability.
I would love to see a feature where it allocates more feeds to a shard
whose machine has bigger capability and lesser feeds whose allocated
machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have
what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks
Vineeth

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hello Alex ,

Using shard allocation , i might be able to move a shard from a week
machine to a string machine.
But this wont fully exploit the capability of the entire network.

What i am looking for is to do the following -
Suppose there are 4 shards on 4 different
machines. I want ES to put 40% of the total data in shard1 , 10% in shard2
, 30% in shard3 and 20% in shard4.

Is this possible in ES ?
If not , wont it be a good idea to post a feature requirement based on this.

Thanks
Vineeth

On Thu, May 30, 2013 at 12:25 PM, Alexander Reelsen alr@spinscale.dewrote:

Hey,

have you taken a look at shard allocation?
See http://www.elasticsearch.org/guide/reference/index-modules/allocation/

It's not automatic (there is no reliable mechanism deciding for
elasticsearch that this node is extraordinary powerful. Judging from its
hardware might be a mistake as many other services might run on the node,
you have to jump in here) mechanism, but you can do it by grouping nodes
for example.

--Alex

On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan vineethmohan@algotree.comwrote:

Hi ,

Is there any provision in elasticsearch where it automatically decides on
increasing or decreasing percentage of feed in a particular shard due to
its hardware capability.
I would love to see a feature where it allocates more feeds to a shard
whose machine has bigger capability and lesser feeds whose allocated
machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should
have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks

       Vineeth

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I don't think it's doable easily.
I mean that you will need to control that on your end and change a little bit the routing feature.

Because, when you want to do a GET, ES is going directly to the right shard using this routing.
How to find the relevant shard if it's not based on id or routing key?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 30 mai 2013 à 11:16, Vineeth Mohan vineethmohan@algotree.com a écrit :

Hello Alex ,

Using shard allocation , i might be able to move a shard from a week machine to a string machine.
But this wont fully exploit the capability of the entire network.

What i am looking for is to do the following -
Suppose there are 4 shards on 4 different machines. I want ES to put 40% of the total data in shard1 , 10% in shard2 , 30% in shard3 and 20% in shard4.

Is this possible in ES ?
If not , wont it be a good idea to post a feature requirement based on this.

Thanks
Vineeth

On Thu, May 30, 2013 at 12:25 PM, Alexander Reelsen alr@spinscale.de wrote:
Hey,

have you taken a look at shard allocation?
See http://www.elasticsearch.org/guide/reference/index-modules/allocation/

It's not automatic (there is no reliable mechanism deciding for elasticsearch that this node is extraordinary powerful. Judging from its hardware might be a mistake as many other services might run on the node, you have to jump in here) mechanism, but you can do it by grouping nodes for example.

--Alex

On Thu, May 30, 2013 at 7:09 AM, Vineeth Mohan vineethmohan@algotree.com wrote:
Hi ,

Is there any provision in elasticsearch where it automatically decides on increasing or decreasing percentage of feed in a particular shard due to its hardware capability.
I would love to see a feature where it allocates more feeds to a shard whose machine has bigger capability and lesser feeds whose allocated machines have lesser hardware capability.

It would be even great if i can tell manually which all shards should have what percentage of total feeds.
Is there any configuration which will help in any of the above.

Thanks

       Vineeth

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.