Can anyone kindly point out why I could be seeing bulk indexing apparently
succeeding on a client but my document count being always zero? I am using
the BulkProcessor API as shown below. If I insert documents individually
via curl, the index is properly updated. I also see all my bulk insert
threads on the server busy and the bulk queue hovering around 150, yet the
filesystem is virtually idle, I see no docs being inserted and no growth on
the index data files. Tried flushing and refreshing manually to no avail.
The code below spins happily reporting no errors. Any help is greatly
appreciated.
class CustomBulkListener extends BulkProcessor.Listener {
override def beforeBulk (executionId:Long, request: BulkRequest)
= {
logger.info("Before bulk: {} actions", request.numberOfActions)
}
override def afterBulk (executionId:Long, request: BulkRequest,
response: BulkResponse) = {
logger.info("After bulk: {} actions", request.numberOfActions)
}
override def afterBulk (executionId:Long, request: BulkRequest,
failure: Throwable) = {
logger.warn("Error executing bulk", failure)
}
}
val bulkProcessor: BulkProcessor = BulkProcessor.
builder(myTransportClient, new CustomBulkListener).
setBulkActions(1000).
setBulkSize(new ByteSizeValue(1, ByteSizeUnit.GB)).
setFlushInterval(TimeValue.timeValueSeconds(5)).
setConcurrentRequests(30).build
var bulkRequest = targetEsClient.prepareBulk
val source = """{"some":"test data"}"""
val totalTuples = 10*1000*1000
logger.info("Starting insert loop for $totalTuples docs")
for (i <- 1 to totalTuples) {
val indexRequest = new IndexRequest(Parms.targetIndex,
Parms.docType).source(source)
bulkProcessor.add(indexRequest)
if (i % 1000 == 0) logger.info(s"At $i docs")
}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c71ef1f5-294f-4693-b288-0263325e1c37%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.