Nest: Bulk insert and timeout

Elastic Stack Elasticsearch
1

Hi guys,

We are using new server with 24 cores, RAID 5, 64 GB RAM
Have about 70 indices and trying to write data to it using bulk insert(batch size 5k per index)
But sometimes we have timeouts:

Invalid NEST response built from a unsuccessful low level call on POST: /index1/message/_bulk # Audit trail of this API call: - [1] BadResponse: Node: http://server:9200/ Took: 00:01:28.1294390 - [2] MaxTimeoutReached: Took: -736221.10:34:34.9317145 # OriginalException: System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

Could you please explain which timeout we reached and can we tune it?

Thanks

2

70 indices, or 70 shards in one index?

The default request timeout for the client is set to 1 minute and can be changed globally in configuration:

var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var connectionSettings = new ConnectionSettings(pool)
        // set the request timeout to 3 minutes
    .RequestTimeout(TimeSpan.FromMinutes(3));

Alternatively, you can set a request timeout on a per request basis, so for example, set a longer one on your bulk requests

client.Bulk(b => b
    .CreateMany(people)
    .RequestConfiguration(r => r
        .RequestTimeout(TimeSpan.FromMinutes(2))
    )
);
3

Hi Russ,

70 indices, each has 1 shard

I guess the issue related to reason we are using spinned HDD
We changed config to:

index.merge.scheduler.max_thread_count: 1
http.max_content_length: 256mb
index.number_of_shards: 1

threadpool.bulk.size: 600
threadpool.bulk.queue_size: 5000

index.refresh_interval: 30s
index.translog.flush_threshold_ops: 500000
indices.memory.index_buffer_size: 30%

and it helped.

Thanks