Missing documents after bulk indexing


(Israel Klein) #1

Hi,

I have a problem for a while now, that I cannot find an explanation for it
so far.
I have a 10 nodes cluster with replication factor of 2 and 10 shards. On
each of the nodes, I have another JVM running my code, using transport
client, sending bulk indexing+delete requests. Sometimes, for a very small
number of the bulk requests, I get and response with errors for some of the
docs in the bulk with error "EsRejectedExecutionException". For these items
only, I create another bulk requests and sending them again. The problem is
that for some reason, a small number (something like 3000 out of 10,000,000
were not indexed and another 40 were not deleted) of the total docs are
"missing"; meaning that they are not indexed or deleted and I did not get
them marked as errors in the responses.
I log all errors and the only ones I get are those
"EsRejectedExecutionException" which I retry.
I see no errors in the ES log files.

Thanks,

Israel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8e9f67d1-3842-44b2-8598-645ce233f6b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #2

What are the details of that exception? can it be that ES has issues
parsing the docs?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, May 19, 2014 at 10:23 PM, Israel Klein israel.klein@gmail.comwrote:

Hi,

I have a problem for a while now, that I cannot find an explanation for it
so far.
I have a 10 nodes cluster with replication factor of 2 and 10 shards. On
each of the nodes, I have another JVM running my code, using transport
client, sending bulk indexing+delete requests. Sometimes, for a very small
number of the bulk requests, I get and response with errors for some of the
docs in the bulk with error "EsRejectedExecutionException". For these items
only, I create another bulk requests and sending them again. The problem is
that for some reason, a small number (something like 3000 out of 10,000,000
were not indexed and another 40 were not deleted) of the total docs are
"missing"; meaning that they are not indexed or deleted and I did not get
them marked as errors in the responses.
I log all errors and the only ones I get are those
"EsRejectedExecutionException" which I retry.
I see no errors in the ES log files.

Thanks,

Israel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8e9f67d1-3842-44b2-8598-645ce233f6b6%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtxUEYG-fBEqXprNFf97JkEGYQBFQ97zZ0DTmJRq4fX-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Israel Klein) #3

That exception means that the indexing/deleting queue is full thus these
documents are rejected.
I have a queue where I take the docs from and then send to ES for indexing.
all rejected are getting back into the queue. At the ends of the processes
the queue is empty !!

On Monday, May 19, 2014 11:50:16 PM UTC+3, Itamar Syn-Hershko wrote:

What are the details of that exception? can it be that ES has issues
parsing the docs?

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Mon, May 19, 2014 at 10:23 PM, Israel Klein <israel...@gmail.com<javascript:>

wrote:

Hi,

I have a problem for a while now, that I cannot find an explanation for
it so far.
I have a 10 nodes cluster with replication factor of 2 and 10 shards. On
each of the nodes, I have another JVM running my code, using transport
client, sending bulk indexing+delete requests. Sometimes, for a very small
number of the bulk requests, I get and response with errors for some of the
docs in the bulk with error "EsRejectedExecutionException". For these items
only, I create another bulk requests and sending them again. The problem is
that for some reason, a small number (something like 3000 out of 10,000,000
were not indexed and another 40 were not deleted) of the total docs are
"missing"; meaning that they are not indexed or deleted and I did not get
them marked as errors in the responses.
I log all errors and the only ones I get are those
"EsRejectedExecutionException" which I retry.
I see no errors in the ES log files.

Thanks,

Israel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8e9f67d1-3842-44b2-8598-645ce233f6b6%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23ad3027-ec2c-4023-9176-636596a39b2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4