Indexing performance drops when indexing a lot of nested documents

Elastic Stack Elasticsearch
1

Hi,

I'm using Elasticsearch to index documents with a number of nested documents in it.
This number varies from a single nested document up to a hundred.
The nested document consists of 13 fields, two of which are multivalued.
What I observe is that as soon as I remove one of the two, indexing performance is just fine.
However when added, indexing performance drops significantly, because the size of the documents is increasing too.

To communicate with my elasticsearch cluster I use the TransportClient and I index documents in bulk using the BulkProcessor.

Current settings for the TransportClient:

  • sniff: false
  • ping_timeout: 60 seconds
  • nodes_sampler_interval: 60 seconds

Current settings for the BulkProcessor:

  • BulkActions: 5000
  • BulkSize: 20 MB
  • ConcurrentRequests: 4

Without the multivalued field, indexing is done in 200 seconds.
With the multivalued field, indexing is done in 2500 seconds.

I'm monitoring for each request the number of documents and the size. Furthermore, I'm monitoring the response times of the Bulkprocessor. This is on average 4 seconds.

This is the what I see in the monitoring plugin when I'm indexing:

Pasted image1774×191 27.9 KB

Does anyone have an idea how to increase indexing performance in this kind of setup?

Thanks,
Robin

2

Topic can be closed.
Issue was not with Elasticsearch, but in indexing application itself.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.