Percolator requests return inconsistent/empty results

Hi,

We're experience an issue where calls to _percolate return inconsistent
results based on which node we hit. Unfortunately I don't have
a reproducible case, this seems to occur intermittently but I can't
diagnose why. Anyone who has experienced this/can suggest debug steps is
appreciated.

Here's an example of what we see:
~$ curl -XGET 'node1.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":["some-result"]}
~$ curl -XGET 'node2.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":[]}

Furthermore, we'll see for the "docs" entry in the below calls:
~$ curl -XGET endpoint.node1:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 8, "deleted_docs" : 0 }
~$ curl -XGET endpoint.node2:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 11, "deleted_docs" : 3 }

Some notes on the occurrence:

  • We're running 5 nodes, and we will see "matches": [] across some number
    of them (2 during the most recent event).
  • The results returned from each node are consistent, meaning that we will
    receive empty matches repeatedly from the same bad node.
  • Flushing or refreshing do not resolve this.

Resolution:
The only way we can resolve this at the moment is by deleting the
_percolator index and re-submitting entries to it.

I'm happy to dig into this further if anyone has alternative debugging
steps, otherwise I'll likely file a bug.

This is running on Ubuntu with java version 1.6.0_27. We have 5 nodes, 3
indices. ES: 0.20.4 with s3 as the gateway.type.

Thanks
oli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I've filed a bug for
this: https://github.com/elasticsearch/elasticsearch/issues/3121
I believe it's a duplicate
of: https://github.com/elasticsearch/elasticsearch/issues/2531 which has
been alive for a while but is unresolved.

As an aside, I also noticed that the files on disk on each of the machines
machines:
/location/elasticsearch/nodes/0/indices/_percolator/0/index/_0.fdt
had the same content.

I added this information to the ticket.

  • oli

On Thursday, May 16, 2013 7:51:55 PM UTC-7, Oli wrote:

Hi,

We're experience an issue where calls to _percolate return inconsistent
results based on which node we hit. Unfortunately I don't have
a reproducible case, this seems to occur intermittently but I can't
diagnose why. Anyone who has experienced this/can suggest debug steps is
appreciated.

Here's an example of what we see:
~$ curl -XGET 'node1.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":["some-result"]}
~$ curl -XGET 'node2.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":[]}

Furthermore, we'll see for the "docs" entry in the below calls:
~$ curl -XGET endpoint.node1:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 8, "deleted_docs" : 0 }
~$ curl -XGET endpoint.node2:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 11, "deleted_docs" : 3 }

Some notes on the occurrence:

  • We're running 5 nodes, and we will see "matches": [] across some number
    of them (2 during the most recent event).
  • The results returned from each node are consistent, meaning that we will
    receive empty matches repeatedly from the same bad node.
  • Flushing or refreshing do not resolve this.

Resolution:
The only way we can resolve this at the moment is by deleting the
_percolator index and re-submitting entries to it.

I'm happy to dig into this further if anyone has alternative debugging
steps, otherwise I'll likely file a bug.

This is running on Ubuntu with java version 1.6.0_27. We have 5 nodes, 3
indices. ES: 0.20.4 with s3 as the gateway.type.

Thanks
oli

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.