Problems expanding replicas for _percolator

Kenneth_Loafman · February 3, 2012, 4:26pm

Hi,

I gave up on my previous question about 0-all being answered, but I'm still
having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Berkay_Mollamustafao · February 3, 2012, 7:08pm

Gist link below comes up as an empty page at least for me. just fyi..

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman
kenneth.loafman@gmail.comwrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Kenneth_Loafman_2 · February 3, 2012, 7:29pm

Sorry, I posted the public git repo link. Try this

gist.github.com

https://gist.github.com/kwloafman/1730911

health

{
  "cluster_name" : "percolate",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 3,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,

This file has been truncated. show original

results

{
  "_percolator" : {
    "settings" : {
      "index.number_of_shards" : "1",
      "index.number_of_replicas" : "4",
      "index.auto_expand_replicas" : "false"
    }
  }
}
{"ok":true}

This file has been truncated. show original

set-replicas.sh

#!/bin/bash

curl -XGET rogue:9200/_percolator/_settings?pretty=true
echo
curl -XPUT rogue:9200/_percolator/_settings -d '
{
    "index" : {
        "number_of_replicas" : 6,
        "auto_expand_replicas" : false
    }

This file has been truncated. show original

...Ken

On Fri, Feb 3, 2012 at 1:08 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

Gist link below comes up as an empty page at least for me. just fyi..

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman <
kenneth.loafman@gmail.com> wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Kenneth_Loafman · February 3, 2012, 11:56pm

Sorry, I posted the public git repo link. Try this

gist.github.com

https://gist.github.com/kwloafman/1730911

health

{
  "cluster_name" : "percolate",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 3,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,

This file has been truncated. show original

results

{
  "_percolator" : {
    "settings" : {
      "index.number_of_shards" : "1",
      "index.number_of_replicas" : "4",
      "index.auto_expand_replicas" : "false"
    }
  }
}
{"ok":true}

This file has been truncated. show original

set-replicas.sh

#!/bin/bash

curl -XGET rogue:9200/_percolator/_settings?pretty=true
echo
curl -XPUT rogue:9200/_percolator/_settings -d '
{
    "index" : {
        "number_of_replicas" : 6,
        "auto_expand_replicas" : false
    }

This file has been truncated. show original

...Ken

On Fri, Feb 3, 2012 at 1:29 PM, Kenneth Loafman kenneth@loafman.com wrote:

Sorry, I posted the public git repo link. Try this
Problem expanding replicas for _percolator · GitHub

...Ken

On Fri, Feb 3, 2012 at 1:08 PM, Berkay Mollamustafaoglu <mberkay@gmail.com

wrote:

Gist link below comes up as an empty page at least for me. just fyi..

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman <
kenneth.loafman@gmail.com> wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of
percolation power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Berkay_Mollamustafao · February 4, 2012, 1:16am

ES does not put 2 copies of the same shard on the same node. You have 3
nodes so you can currently have max 2 replicas. It's probably why your
cluster status is yellow as well.

On Friday, February 3, 2012, Kenneth Loafman kenneth@loafman.com wrote:

Sorry, I posted the public git repo link. Try this
Problem expanding replicas for _percolator · GitHub
...Ken

On Fri, Feb 3, 2012 at 1:08 PM, Berkay Mollamustafaoglu mberkay@gmail.com
wrote:

Gist link below comes up as an empty page at least for me. just fyi..
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman <
kenneth.loafman@gmail.com> wrote:

Hi,
I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.
I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,
increasing the number of replicas will increase the number of
percolation power,
but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.
git://gist.github.com/1730911.git
...Thanks,
...Ken

--
Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292
mberkay on yahoo, google and skype

Kenneth_Loafman_2 · February 4, 2012, 2:27am

So how do you get more replicas? Should I run another node on each machine?

On Fri, Feb 3, 2012 at 7:16 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

ES does not put 2 copies of the same shard on the same node. You have 3
nodes so you can currently have max 2 replicas. It's probably why your
cluster status is yellow as well.

On Friday, February 3, 2012, Kenneth Loafman kenneth@loafman.com wrote:

Sorry, I posted the public git repo link. Try this
Problem expanding replicas for _percolator · GitHub
...Ken

On Fri, Feb 3, 2012 at 1:08 PM, Berkay Mollamustafaoglu <
mberkay@gmail.com> wrote:

Gist link below comes up as an empty page at least for me. just fyi..
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman <
kenneth.loafman@gmail.com> wrote:

Hi,
I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.
I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,
increasing the number of replicas will increase the number of
percolation power,
but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.
git://gist.github.com/1730911.git
...Thanks,
...Ken

--
Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292

mberkay on yahoo, google and skype

Kenneth_Loafman · February 4, 2012, 5:30pm

Thanks. I did not understand that limitation.

I solved the problem by adding a 2nd node to each machine to get more
percolators (now a 6-node cluster). That lowered the average time per
percolate, so problem is solved, for now.

...Ken

On Fri, Feb 3, 2012 at 7:16 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

ES does not put 2 copies of the same shard on the same node. You have 3
nodes so you can currently have max 2 replicas. It's probably why your
cluster status is yellow as well.

On Friday, February 3, 2012, Kenneth Loafman kenneth@loafman.com wrote:

Sorry, I posted the public git repo link. Try this
Problem expanding replicas for _percolator · GitHub
...Ken

On Fri, Feb 3, 2012 at 1:08 PM, Berkay Mollamustafaoglu <
mberkay@gmail.com> wrote:

Gist link below comes up as an empty page at least for me. just fyi..
Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Fri, Feb 3, 2012 at 11:26 AM, Kenneth Loafman <
kenneth.loafman@gmail.com> wrote:

Hi,
I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.
I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,
increasing the number of replicas will increase the number of
percolation power,
but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.
git://gist.github.com/1730911.git
...Thanks,
...Ken

--
Regards,
Berkay Mollamustafaoglu
Ph: +1 (571) 766-6292

mberkay on yahoo, google and skype

kimchy · February 5, 2012, 5:11pm

The comment is meant to say that increasing the number of replicas for the actual index (not _percoalte, which just acts as the storage to the registered queries) will increase percolation performance. The _percolate index should not change, and should always be 1 shard with 0-all replicas (so it will be spread across the nodes storage wise, and all nodes will be able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the documentation,

increasing the number of replicas will increase the number of percolation power,

but I'm having trouble getting the number of replicas to increase. I played with the script in the gist below, but the results are the same, every time. The cluster is showing unassigned shards and they never get initialized. Please help.

git://gist.github.com/1730911.git (Problem expanding replicas for _percolator · GitHub)

...Thanks,
...Ken

Kenneth_Loafman_2 · February 5, 2012, 5:19pm

As it turns out the indexes are fake ('filters' and 'excludes') with the
same mapping as the other indexes and these reside on the percolator
cluster, so my solution worked, but for a different reason than I thought.
They are also 1 shard, 0-all replicas. So I guess the only way to
increase the number of filters and excludes indexes is to add nodes, if I
understand correctly?

On Sun, Feb 5, 2012 at 11:11 AM, Shay Banon kimchy@gmail.com wrote:

The comment is meant to say that increasing the number of replicas for
the actual index (not _percoalte, which just acts as the storage to the
registered queries) will increase percolation performance. The _percolate
index should not change, and should always be 1 shard with 0-all replicas
(so it will be spread across the nodes storage wise, and all nodes will be
able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Kenneth_Loafman · February 5, 2012, 5:22pm

As it turns out the indexes are fake ('filters' and 'excludes') with the
same mapping as the other indexes and these reside on the percolator
cluster, so my solution worked, but for a different reason than I thought.
They are also 1 shard, 0-all replicas. So I guess the only way to
increase the number of filters and excludes indexes is to add nodes, if I
understand correctly?

On Sun, Feb 5, 2012 at 11:11 AM, Shay Banon kimchy@gmail.com wrote:

The comment is meant to say that increasing the number of replicas for
the actual index (not _percoalte, which just acts as the storage to the
registered queries) will increase percolation performance. The _percolate
index should not change, and should always be 1 shard with 0-all replicas
(so it will be spread across the nodes storage wise, and all nodes will be
able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

kimchy · February 5, 2012, 5:38pm

What do you mean by fake indices? You mean they hold no data except for percolating against them? Thats fine, they are still your "compute" power for percolation (and sometimes, they also hold data).

Note, once a shard for an index (not the _percolate index, the one you register the query against) exists on a node, then it will make use of that node. The percolation action can be executed concurrently on the same shard. And yes, adding more nodes will mean better performance.

With percolation, if you have "fake" indices as you mentioned, then its good enough to have a single shard of those fake indices, and 0-all for the replicas. This will mean that automatically, as you add nodes, those replicas will expand to make use of the new node, and will also service percolation requests.

Still, not to be confused with the _percolate index, which only acts as persistent storage for the percolated queries, and needs to be 0-all so it will exists on all nodes.

On Sunday, February 5, 2012 at 7:22 PM, Kenneth Loafman wrote:

As it turns out the indexes are fake ('filters' and 'excludes') with the same mapping as the other indexes and these reside on the percolator cluster, so my solution worked, but for a different reason than I thought. They are also 1 shard, 0-all replicas. So I guess the only way to increase the number of filters and excludes indexes is to add nodes, if I understand correctly?

On Sun, Feb 5, 2012 at 11:11 AM, Shay Banon <kimchy@gmail.com (mailto:kimchy@gmail.com)> wrote:

The comment is meant to say that increasing the number of replicas for the actual index (not _percoalte, which just acts as the storage to the registered queries) will increase percolation performance. The _percolate index should not change, and should always be 1 shard with 0-all replicas (so it will be spread across the nodes storage wise, and all nodes will be able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the documentation,

increasing the number of replicas will increase the number of percolation power,

but I'm having trouble getting the number of replicas to increase. I played with the script in the gist below, but the results are the same, every time. The cluster is showing unassigned shards and they never get initialized. Please help.

git://gist.github.com/1730911.git (Problem expanding replicas for _percolator · GitHub)

...Thanks,
...Ken

Kenneth_Loafman_2 · February 5, 2012, 6:45pm

Yes, the fake indexes contain no data. I add an extra field to each filter
which give me information on what matched and where to store it. That
works out well for a multi-tenant operation where there may be overlap in
the filters.

On Sun, Feb 5, 2012 at 11:38 AM, Shay Banon kimchy@gmail.com wrote:

What do you mean by fake indices? You mean they hold no data except for
percolating against them? Thats fine, they are still your "compute" power
for percolation (and sometimes, they also hold data).

Note, once a shard for an index (not the _percolate index, the one you
register the query against) exists on a node, then it will make use of that
node. The percolation action can be executed concurrently on the same
shard. And yes, adding more nodes will mean better performance.

With percolation, if you have "fake" indices as you mentioned, then its
good enough to have a single shard of those fake indices, and 0-all for the
replicas. This will mean that automatically, as you add nodes, those
replicas will expand to make use of the new node, and will also service
percolation requests.

Still, not to be confused with the _percolate index, which only acts as
persistent storage for the percolated queries, and needs to be 0-all so it
will exists on all nodes.

On Sunday, February 5, 2012 at 7:22 PM, Kenneth Loafman wrote:

As it turns out the indexes are fake ('filters' and 'excludes') with the
same mapping as the other indexes and these reside on the percolator
cluster, so my solution worked, but for a different reason than I thought.
They are also 1 shard, 0-all replicas. So I guess the only way to
increase the number of filters and excludes indexes is to add nodes, if I
understand correctly?

On Sun, Feb 5, 2012 at 11:11 AM, Shay Banon kimchy@gmail.com wrote:

The comment is meant to say that increasing the number of replicas for
the actual index (not _percoalte, which just acts as the storage to the
registered queries) will increase percolation performance. The _percolate
index should not change, and should always be 1 shard with 0-all replicas
(so it will be spread across the nodes storage wise, and all nodes will be
able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken

Kenneth_Loafman · February 5, 2012, 6:45pm

Yes, the fake indexes contain no data. I add an extra field to each filter
which give me information on what matched and where to store it. That
works out well for a multi-tenant operation where there may be overlap in
the filters.

On Sun, Feb 5, 2012 at 11:38 AM, Shay Banon kimchy@gmail.com wrote:

What do you mean by fake indices? You mean they hold no data except for
percolating against them? Thats fine, they are still your "compute" power
for percolation (and sometimes, they also hold data).

Note, once a shard for an index (not the _percolate index, the one you
register the query against) exists on a node, then it will make use of that
node. The percolation action can be executed concurrently on the same
shard. And yes, adding more nodes will mean better performance.

With percolation, if you have "fake" indices as you mentioned, then its
good enough to have a single shard of those fake indices, and 0-all for the
replicas. This will mean that automatically, as you add nodes, those
replicas will expand to make use of the new node, and will also service
percolation requests.

Still, not to be confused with the _percolate index, which only acts as
persistent storage for the percolated queries, and needs to be 0-all so it
will exists on all nodes.

On Sunday, February 5, 2012 at 7:22 PM, Kenneth Loafman wrote:

As it turns out the indexes are fake ('filters' and 'excludes') with the
same mapping as the other indexes and these reside on the percolator
cluster, so my solution worked, but for a different reason than I thought.
They are also 1 shard, 0-all replicas. So I guess the only way to
increase the number of filters and excludes indexes is to add nodes, if I
understand correctly?

On Sun, Feb 5, 2012 at 11:11 AM, Shay Banon kimchy@gmail.com wrote:

The comment is meant to say that increasing the number of replicas for
the actual index (not _percoalte, which just acts as the storage to the
registered queries) will increase percolation performance. The _percolate
index should not change, and should always be 1 shard with 0-all replicas
(so it will be spread across the nodes storage wise, and all nodes will be
able to read the queries registered).

On Friday, February 3, 2012 at 6:26 PM, Kenneth Loafman wrote:

Hi,

I gave up on my previous question about 0-all being answered, but I'm
still having troubles with expanding the number of replicas for _percolator.

I'm running 18.6 on 3 nodes, strictly for percolator use. Per the
documentation,

increasing the number of replicas will increase the number of percolation
power,

but I'm having trouble getting the number of replicas to increase. I
played with the script in the gist below, but the results are the same,
every time. The cluster is showing unassigned shards and they never get
initialized. Please help.

git://gist.github.com/1730911.git

...Thanks,
...Ken