The function doAccept() is invoked repeatedly from my application and in each invocation the function inserts a single entry into ES. However, this results in certain entries getting dropped (more specifically, deleted from ES). For example, if I call this function 1000 times (i.e. in total for 1000 entries), only 800 or so (some random number) get inserted finally into ES.
However, adding a Thread.sleep(100) (basically introducing a delay before adding an entry) at the beginning of the function solves this issue.
The function doAccept() is invoked repeatedly from my application and in
each invocation the function inserts a single entry into ES. However, this
results in certain entries getting dropped (more specifically, deleted from
ES). For example, if I call this function 1000 times (i.e. in total for
1000 entries), only 800 or so (some random number) get inserted finally
into
ES.
However, adding a Thread.sleep(100) (basically introducing a delay before
adding an entry) at the beginning of the function solves this issue.
You are using System.currentTimeMillis() to generate the id value.
ES is so fast at executing your index request that you are getting duplicate
ids. :^)
I say this slightly tongue in cheek because I would expect at least a milli
to tick off the clock before the id is generated again, especially since you
are using the actionGet() method which basically turns this into a
synchronous call.
If the request is getting a duplicate id, then the version field
should have been incremented. One question I have always been meaning
to ask is if it is possible to query based on the version field?
Simple queries such as give me all documents with a version greater or
equal to 2.
I think the problem is right in the code.
You are using System.currentTimeMillis() to generate the id value.
ES is so fast at executing your index request that you are getting duplicate
ids. :^)
I say this slightly tongue in cheek because I would expect at least a milli
to tick off the clock before the id is generated again, especially since you
are using the actionGet() method which basically turns this into a
synchronous call.
-- jim
Ha!, I missed that part :), good catch James. Regarding millisecond
resolution, ES can definitely index data faster than 1 millisecond
(depending on the complexity of the indexed doc), but also, note that
System.currentTimeInMillis does not guarantee 1 millisecond resolution.
You are using System.currentTimeMillis() to generate the id value.
ES is so fast at executing your index request that you are getting
duplicate ids. :^)
I say this slightly tongue in cheek because I would expect at least a milli
to tick off the clock before the id is generated again, especially since you
are using the actionGet() method which basically turns this into a
synchronous call.
If once does not provide a version when indexing data, then the doc is
simply updated. A version check is performed on index operation only when
the version is provided. If, on the other hand, a create was used (and not
index), then there would have been failures because some documents would
have already existed.
There isn't an option to ask for all documents with a version greater than
something. I guess that its meaningful when providing an external version,
in which case, that value can be simply also used as an additional field in
the doc. Though, it can possibly be automatically done in es.
On Thu, Aug 25, 2011 at 4:38 PM, Ivan Brusic ivan@brusic.com wrote:
If the request is getting a duplicate id, then the version field
should have been incremented. One question I have always been meaning
to ask is if it is possible to query based on the version field?
Simple queries such as give me all documents with a version greater or
equal to 2.
I think the problem is right in the code.
You are using System.currentTimeMillis() to generate the id value.
ES is so fast at executing your index request that you are getting
duplicate
ids. :^)
I say this slightly tongue in cheek because I would expect at least a
milli
to tick off the clock before the id is generated again, especially since
you
are using the actionGet() method which basically turns this into a
synchronous call.
-- jim
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.