Elasticsearch is taking too long between a BulkImport and another

Elastic Stack Elasticsearch
1

Hi all,

we have a microservice application that is getting data from Kafka and importing them into Elasticsearch using BulkImport operation.

  • The microservice application is running in docker using docker-compose and scaling for parallel and multi-thread.
  • Elasticsearch (2.4.1) is also running into docker using docker-compose with the following configuration (1master with 4GB javaHeapSize - 1client with 4GB javaHeapSize - 5data with 8GB javaHeapSize - 24shards - 1index~7.89GB).
  • The VM have 256GB of RAM, 24 CPU (24core), 500GB disk space ext4

We noted that the application is taking 20s between some BulkImport and continuing with others, at the end to import a fullIndex of 7.49GB (6,2Milions hits) is taking 4h40m.. Not what we expected.

We already tried to:

  1. Disable refresh and replicas for initial loads
  2. Setting ulimits higher
  3. Setting scale configuration of threadpools

No luck.
Can we have some suggestion in order to increase indexing speed?

2

I would recommend looking at this guide. You may also want to optimise your mappings to reduce work required at indexing time.

Having a single master-eligible node makes it a single point of failure and is not recommended. Also make sure you are sending data directly to the data nodes so the client node does not become a bottleneck.

3

Thanks for the fast reply @Christian_Dahlqvist, this env is just a prototype in order to get some metrics useful for prod, were we have 3 clients, 3 master, 12 data nodes..
Anyway good point!
Do you know if client node is limiting number of requests/rate?

4

Have a look at the Elasticsearch nodes and see if you can identify what is limiting throughput. Elasticsearch is often very disk I/O intensive, so slow storage is a common bottleneck, but it could also be CPU or GC.

5

Is not for sure CPU and GC, I already monitored it and it's ok.. Regarding disk I/O we are using volumes with default docker driver.

6

What kind of storage do you have?

7

VM is running in VMWare vCloud Director with the so called "Fast Storage with Snapshot Site B2"..

8

I have no idea what that means or corresponds to.

9

Sorry me neither.. I have no more detailed information about this Cloud Storage.

10

Look at iostat on the VM while it is under load, and see if that gives any indication.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.