Missing in bulk response

Elastic Stack Elasticsearch
1

Hi,
Often times I observe that Java BulkItemResponse does not contain some of documents I put in the corresponding BulkRequest.
I don't see any error log in ES's log file.
(I have action: DEBUG in logging.yml)

Please guide me where to look at.
Thank you for your help.

2

I found on the client side, I have following error:
ERROR; ... bulk request:org.elasticsearch.ElasticSearchTimeoutException: Timeout waiting for task.

This must be the reason why I don't see any document ids in the BulkResponse.

I don't see any corresponding errors/infos in any of elasticsearch data nodes.
(my client is not a data node. it is a Java client node).

It is happening now and this time the error is only from one of my indices.
(I issue bulk requests by index. I have 70+ indices.)

The other day, I saw the problem from some (but not all) indices.

Please help me where I should look at to investigate further.
Thank you.

3

Hiya

On Wed, 2012-10-10 at 14:46 -0700, arta wrote:

I found on the client side, I have following error:
ERROR;[search.analyzer.ElasticSearchIndexer:303][BatchingQueue-timer] bulk
request:org.elasticsearch.ElasticSearchTimeoutException: Timeout waiting for
task.

How many documents (and of what size) are you bulk indexing in a single
request?

It is happening now and this time the error is only from one of my indices.
(I bulk requests by index. I have 70+ indices.)

70 indices running on how many nodes?

My guess would be:

  1. passing too much data at once to bulk
  2. running too many indices on too few nodes with not enough resources

clint

--

4

Also, it seems like the timeout is not coming from elasticsearch itself, but from the client library you are using? If its the Java client, do you execute the bulk with a timeout?

On Oct 11, 2012, at 12:46 AM, Clinton Gormley clint@traveljury.com wrote:

Hiya

On Wed, 2012-10-10 at 14:46 -0700, arta wrote:

I found on the client side, I have following error:
ERROR;[search.analyzer.ElasticSearchIndexer:303][BatchingQueue-timer] bulk
request:org.elasticsearch.ElasticSearchTimeoutException: Timeout waiting for
task.

How many documents (and of what size) are you bulk indexing in a single
request?

It is happening now and this time the error is only from one of my indices.
(I bulk requests by index. I have 70+ indices.)

70 indices running on how many nodes?

My guess would be:

  1. passing too much data at once to bulk
  2. running too many indices on too few nodes with not enough resources

clint

--

--

5

Thank you for the replies, Clinton, Kimchy.

Here are answers to your questions:

How many documents (and of what size) are you bulk indexing in a single request?

I limit up to 100 indexings or deletions in a bulk.

70 indices running on how many nodes?

I have 13 nodes.

Also, it seems like the timeout is not coming from elasticsearch itself, but from the client library you are using? If its the Java client, do you execute the bulk with a timeout?

The exception is caught by my code. It is thrown by the following code:
BulkRequestBuilder req = client.prepareBulk();
req.add(...); // up to 100 times
req.execute().actionGet(60000, TimeUnit.MILLISECONDS);