Cannot found inserted document after delete index

Elastic Stack Elasticsearch
1

I write a simple test validate duplicates are not exist:

@Test
public void testSameDataNotPushedTwice() throws Exception {
	// Do some logic
	// index contains es index name
	
	// adding this line fail the test
	// deleteOldData(esPersistence.getESClient(), index);
	esPersistence.insert(cdrData);
	esPersistence.insert(cdrData);

	SearchResponse searchResponse = getDataFromElastic(esPersistence.getESClient(), index);
	assertThat(searchResponse.getHits().getHits().length).isEqualTo(1);
}

As you can see I push data to ES and check hits length equals 1.

Test is passed when the delete line is in commnet.

Now, I want to make sure there is no data from old / others tests, therefore I want to delete the index before the insert. The problem: delete and then insert return 0 hits in Search Response.

The delete index method:

public static void deleteOldData(RestHighLevelClient client, String index) throws IOException {
	GetIndexRequest request = new GetIndexRequest(index);
	boolean exists = client.indices().exists(request, RequestOptions.DEFAULT);
	if (exists) {
		DeleteIndexRequest deleteRequest = new DeleteIndexRequest(index);
		client.indices().delete(deleteRequest, RequestOptions.DEFAULT);
	}
}
  • ES 7.6.2
  • The data is exist in ES.
  • Adding sleep not solve the problem (even for 10 seconds).
  • The search is working (document is found) while debbuging.
  • SettingsResponse not return refresh_interval (so the default is 1s)

Bottom line: How can I perform delete index --> insert --> search and found the documents?

More Info:

Insert methods:

public boolean insert(List<ProjectData> projDataList) {
	// Relevant Lines
	BulkRequest bulkRequest = prepareBulkRequests(projDataList, esConfiguration.getCdrDataIndexName());
	insertBulk(bulkRequest)
}

private BulkRequest prepareBulkRequests(List<ProjectData> data, String indexName) {
	BulkRequest bulkRequest = new BulkRequest();
	for (ProjectData ProjectData : data) {
		String json = jsonParser.parsePojo(ProjectData);

		bulkRequest.add(new IndexRequest(indexName)
				.id(ProjectData.getId())
				.source(json, XContentType.JSON));
	}

	return bulkRequest;
}

private boolean insertBulk(BulkRequest bulkRequest) {
	try {
		BulkResponse bulkResponse = rhlClient.bulk(bulkRequest, RequestOptions.DEFAULT);

		if (bulkResponse.hasFailures()) {
			logger.error(buildCustomBulkFailedMessage(bulkResponse));
			return false;
		}

	} catch (IOException e) {
		logger.warn("Failed to insert csv fields. Error: {}", e.getMessage());
		return false;
	}

	return true;
}
2

I think you need to make sure that you are refreshing the index after the bulk execution and before the search operation.

Have a look at:

Might help...

3

Thank you very much!! actually refresh is enough (not need the IndexRequest line in the code)
Thanks again,
Assaf

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.