Java REST Client and Multithreading

[mc1392]

Is the Java REST Client Thread safe?

Can I instantiate a single client object and share among the threads?
Or should I open separate connections in each thread?

[dadoonet (David Pilato)]

One client only is fine IMO.

[mc1392]

7000 records are taking forever to reach Elastic

[dadoonet (David Pilato)]

But what are you doing?

Calling index() 7000 times? If so, that's not the best solution here.
But please be more specific about what your problem is.

[mc1392]

I'm looping, the reason is, the AWS instance I was using kept running out of memory while uploading to Hubspot. Keeping the objects in memory was the problem, and very little RAM.

So I decided to serialize the objects then send to Elastic. This is why I didn't use the BULK API in the first place.

I've since increased the RAM from 2 GB to 8GB, so maybe I can revisit the BULK API.

public static void insertCDSData(HubspotContact registrant) {
	
	CDSRegImport_ES_Credentials credentials = new CDSRegImport_ES_Credentials();
	RestHighLevelClient client = credentials.getClient();
	
	
	/////////////////////////////////////////////////////////////////////////////////////////////////////////
	
		
		
		IndexRequest indexRequest = new IndexRequest("cdsregimport", "CDSREGIMPORT", registrant.getBadge()).
				source("firstname",registrant.getFirstName(), 
						"lastname", registrant.getLastName(),
						"phone",registrant.getPhone(),
						"ATS ID",registrant.getATSID(),
						"email",registrant.getEmail(),
						"company",registrant.getCompany(),
						"degree",registrant.getDegreeDesignation(),
						"Address 1",registrant.getMailStreet1(),
						"Address 2", registrant.getMailStreet2(),
						"City",registrant.getMailCity(),
						"State",registrant.getMailState(),
						"Country", registrant.getMailCountry(),
						"Registration Type", registrant.getRegistrationType(),
						"Title", registrant.getTitle(),
						"Topics of Interest",registrant.getTopicOfInterest(),
						"CDS Address Type",registrant.getAddressType(),
						"Work Settings", registrant.getWorkSettings(),
						"Nature of Professional Activities", registrant.getNatureOfProfessionalWork(),
						"zipcode",registrant.getMailZip(),
						"postDate", new Date());
		UpdateRequest upR = new UpdateRequest("cdsregimport", "CDSREGIMPORT", String.valueOf(registrant.getBadge())).doc(indexRequest)
				.upsert(indexRequest);
		UpdateResponse updateResponse = null;
		
		try{
			updateResponse = client.update(upR);
		}catch(IOException ioe){
			ioe.printStackTrace();
		}
		upR = null;
		registrant = null;
		indexRequest = null;
	
	
	credentials.closeConnection();

}

[dadoonet (David Pilato)]

Definitely the bulk processor will help you here.

BTW did you look at cloud.elastic.co? It runs also on AWS.

[mc1392]

AWS is way too expensive for ES.
Other services, yes, ES, no.