Hi,
We're experience an issue where calls to _percolate return inconsistent
results based on which node we hit. Unfortunately I don't have
a reproducible case, this seems to occur intermittently but I can't
diagnose why. Anyone who has experienced this/can suggest debug steps is
appreciated.
Here's an example of what we see:
~$ curl -XGET 'node1.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":["some-result"]}
~$ curl -XGET 'node2.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":[]}
Furthermore, we'll see for the "docs" entry in the below calls:
~$ curl -XGET endpoint.node1:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 8, "deleted_docs" : 0 }
~$ curl -XGET endpoint.node2:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 11, "deleted_docs" : 3 }
Some notes on the occurrence:
- We're running 5 nodes, and we will see "matches": [] across some number
of them (2 during the most recent event).
- The results returned from each node are consistent, meaning that we will
receive empty matches repeatedly from the same bad node.
- Flushing or refreshing do not resolve this.
Resolution:
The only way we can resolve this at the moment is by deleting the
_percolator index and re-submitting entries to it.
I'm happy to dig into this further if anyone has alternative debugging
steps, otherwise I'll likely file a bug.
This is running on Ubuntu with java version 1.6.0_27. We have 5 nodes, 3
indices. ES: 0.20.4 with s3 as the gateway.type.
Thanks
oli
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
I've filed a bug for
this: Percolator requests return inconsistent/empty results · Issue #3121 · elastic/elasticsearch · GitHub
I believe it's a duplicate
of: Percolator responses become inconsistent with cluster after forceful reboot of a node · Issue #2531 · elastic/elasticsearch · GitHub which has
been alive for a while but is unresolved.
As an aside, I also noticed that the files on disk on each of the machines
machines:
/location/elasticsearch/nodes/0/indices/_percolator/0/index/_0.fdt
had the same content.
I added this information to the ticket.
On Thursday, May 16, 2013 7:51:55 PM UTC-7, Oli wrote:
Hi,
We're experience an issue where calls to _percolate return inconsistent
results based on which node we hit. Unfortunately I don't have
a reproducible case, this seems to occur intermittently but I can't
diagnose why. Anyone who has experienced this/can suggest debug steps is
appreciated.
Here's an example of what we see:
~$ curl -XGET 'node1.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":["some-result"]}
~$ curl -XGET 'node2.endpoint:9200/abc/def/_percolate' -d
'{"doc":{"foo":"bar"}}'
{"ok":true,"matches":}
Furthermore, we'll see for the "docs" entry in the below calls:
~$ curl -XGET endpoint.node1:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 8, "deleted_docs" : 0 }
~$ curl -XGET endpoint.node2:9200/_percolator/_status
"docs" : { "num_docs" : 8, "max_doc" : 11, "deleted_docs" : 3 }
Some notes on the occurrence:
- We're running 5 nodes, and we will see "matches": across some number
of them (2 during the most recent event).
- The results returned from each node are consistent, meaning that we will
receive empty matches repeatedly from the same bad node.
- Flushing or refreshing do not resolve this.
Resolution:
The only way we can resolve this at the moment is by deleting the
_percolator index and re-submitting entries to it.
I'm happy to dig into this further if anyone has alternative debugging
steps, otherwise I'll likely file a bug.
This is running on Ubuntu with java version 1.6.0_27. We have 5 nodes, 3
indices. ES: 0.20.4 with s3 as the gateway.type.
Thanks
oli
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.