How to improve performance of application for inserting list in elasticSearch?

Elastic Stack Elasticsearch
1

I have 2000 entries in my List and trying to insert this list into elasticsearch by API which is nothing but a bulk request.
Please find below code for the same.

for(Car car : carList){
elasticSearchService.addOrganization(car,"Car");
}

@Autowired
private Client esClient;

private void addOrganization(Object object, String modelName){
Gson gson = new Gson();
final String json = gson.toJson(object);
esClient.prepareIndex("ElasticSearchIndex", modelName).setSource(json).execute().actionGet();
}

I have made following entry in elasticsearch.yml file
threadpool:
index:
size: 250
queue_size: 1000

As we are monitoring performance of the application through JMeter, we found that our application generates 2000 HTTP requests to connect and put data in elasticsearch
which takes around 10 seconds for this task.Can we make connectionpool in esClient or is there any way to do configuration at elasticsearch server so that performance of the application can be improved and
response time will be reduced upto 3 seconds?

2

Hi @Vrushali,

These are not bulk requests. You are indexing each document in an explicit request and this kills your performance. You can use BulkProcessor which offers a more high-level API and helps you sending bulk requests. Then your indexing throughput should be much better.

Daniel

1 Like
3

Hi, you are not using the bulk API, here is the way to do it :

		BulkRequestBuilder bulkRequest = esClient.prepareBulk();
		bulkRequest.setRefresh(false);

		for(Car car : carList){
					Gson gson = new Gson();
					final String json = gson.toJson(object);
					IndexRequestBuilder indexRequestBuilder = esClient.prepareIndex("ElasticSearchIndex", modelName).setSource(json).execute().actionGet();

					bulkRequest.add(indexRequestBuilder);
		}
		BulkResponse response = bulkRequest.execute().actionGet();

Xavier