Bulk Processor doesn't auto flush after flushInterval duration

Elastic Stack Elasticsearch
1

Hi there,

I have a problem with bulk indexing using Bulk Processor. I setup like this:
.setBulkActions(5000)
.setBulkSize(new ByteSizeValue(50, ByteSizeUnit.MB))
.setFlushInterval(TimeValue.timeValueSeconds(5))
.setConcurrentRequests(3)
.setBackoffPolicy(BackoffPolicy.exponentialBackoff(TimeValue.timeValueMillis(100), 5))
.build();

In my application, I need to keep bulk processor to listen for new coming index request, then BulkProcessor is not allowed to close (by BulkProcessor.close). However, Bulk Processor doesn't auto flush remaining request after flushInterval duration, see below.

Use BulkProcessor.close, indexed 4156 documents
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test-index Y4Y-HRwpT8SM4Uv3V3v2OA 5 1 4156 0 3.7mb 3.7mb

Without BulkProcessor.close, only indexed 3469 documents
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open test-index J_zoR04SRtKZO4p1c-UAWw 5 1 3469 0 2.3mb 2.3mb

So, I would like to know how to index whole documents while no need to close bulk processor. Any help would be appreciated!

2

Are you stopping your application? Do you run that in the context of a test?

Can you try also with concurrent requests set to 1?

1 Like
3

I just check my code and seems the application stopped right after finish inserting index request to bulk, then the last request batch is ignored. Thank you so much for your suggestion @dadoonet!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.