Rest Client listener timeout after waiting for [30000] ms

Elastic Stack Elasticsearch
1

I am using RestClient bulk api to push excel data to elasticsearch but I am getting
listener timeout after waiting for [30000] ms.
my jar version are
httpasyncclient-4.1.2.jar
httpclient-4.5.2.jar
httpcore-4.4.5.jar
httpcore-nio-4.4.5.jar
rest-5.3.0.jar

Please provide me solution for listener timeout after waiting for [30000] ms

1 Like
2

No idea. I don't know what you are exactly doing...

3

I am reading excel file and pushing data to elasticsearch using RestClient bulk API. It is working fine with localhost
but giving listener timeout exception for remote machine i.e. remote elasticsearch. Sometime data gets push into remote elasticsearch but sometime getting this exception. Please provide solution.

Below is my code.

SimpleDateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
dateFormat.setTimeZone(TimeZone.getTimeZone("UTC"));
RestClient restClient = RestClient.builder(new HttpHost("localhost", 9200, "http")).build();

JSONObject obj = new JSONObject();

FileInputStream file = new FileInputStream(new File("C:\Users\cajitb\Desktop\TempFiles\op\TestExcel.xlsx"));
Workbook workbook = new Workbook(file);
Worksheet worksheet = workbook.getWorksheets().get(0);
Cells cells = worksheet.getCells();

	Range range = cells.getMaxDisplayRange();
	int totalColumns = range.getColumnCount();
	int totalRows = range.getRowCount();
	RowCollection rows = cells.getRows();
    
	int maxColumn = cells.getMaxDataColumn();
	int maxRows = cells.getMaxDataRow();

	SimpleDateFormat format = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
	Date d1 = new Date();
	
	
	List<Object> colList = new ArrayList<Object>();
	String actionMetaData = String.format("{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\" } }%n", "jsonindex", "jsonindextype");
	for(int i = 0; i<=0 ; i++)
	{
		for (int j = 0; j < totalColumns; j++) 
	    {
			colList.add(cells.get(i, j).getValue());
	    }
	}
	List<String> bulkData = new ArrayList<String>();
	for (int i = 1; i < rows.getCount(); i++) 
	{
	    for (int j = 0; j < totalColumns; j++) 
	    {
	    	String key = (String)colList.get(j); 
	    	if(null==key || "".equals(key))
	    		key = "Column"+j;
	    	obj.put(key, cells.get(i, j).getValue());
	       
	    }
	     Date d = new Date();
	    obj.put("Action_Date", dateFormat.format(d));
	    
	    String jsonString  = obj.toJSONString();
	    bulkData.add(jsonString);
		obj.clear();
	}
	
	Date d2 = new Date();
	StringBuilder bulkRequestBody = new StringBuilder();
	for (String bulkItem : bulkData) {
	    bulkRequestBody.append(actionMetaData);
	    bulkRequestBody.append(bulkItem);
	    bulkRequestBody.append("\n");
	}
    HttpEntity entity = new NStringEntity(bulkRequestBody.toString(), ContentType.APPLICATION_JSON);
   Response indexResponse = restClient.performRequest(
			"POST",
			"/jsonindex/jsonindextype/_bulk",
			Collections.<String, String>emptyMap(),
			entity);
			//System.out.println(EntityUtils.toString(indexResponse.getEntity()));
	if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
	{
		System.out.println("Index Successfully Created");
	}
4

Try with less requests within a bulk. But I think I already told you that.

5

I am now pushing half of the data in one request and half of the data in second request. Though I am getting the below exception : listener timeout after waiting for [30000] ms
Below is my code :

try {
		
JSONObject obj = new JSONObject();
		
FileInputStream file = new FileInputStream(new File("C:\\Users\\cajitb\\Desktop\\TempFiles\\op\\TestExcel01.xlsx"));
Workbook workbook = new Workbook(file);
Worksheet worksheet = workbook.getWorksheets().get(0);
Cells cells = worksheet.getCells();
		
		Range range = cells.getMaxDisplayRange();
		int totalColumns = range.getColumnCount();
		int totalRows = range.getRowCount();
		RowCollection rows = cells.getRows();
		int maxColumn = cells.getMaxDataColumn();
		int maxRows = cells.getMaxDataRow();
	
		SimpleDateFormat format = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
		Date d1 = new Date();
		System.out.println("Start Time:"+format.format(d1));
		
		List<Object> colList = new ArrayList<Object>();
		String actionMetaData = String.format("{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\" } }%n", "jsonindex", "jsonindextype");
		for(int i = 0; i<=0 ; i++)
		{
			for (int j = 0; j < totalColumns; j++) 
		    {
				colList.add(cells.get(i, j).getValue());
		    }
		}
		List<String> bulkData = new ArrayList<String>();
		for (int i = 1; i < rows.getCount(); i++) 
		{
		    for (int j = 0; j < totalColumns; j++) 
		    {
		    	String key = (String)colList.get(j); 
		    	if(null==key || "".equals(key))
		    		key = "Column"+j;
		    	obj.put(key, cells.get(i, j).getValue());
		    }
	  
		    Date d = new Date();
		    obj.put("Action_Date", dateFormat.format(d));
		    
		    String jsonString  = obj.toJSONString();
		    bulkData.add(jsonString);
			obj.clear();
		}
		StringBuilder bulkRequestBody = new StringBuilder();
		
		int halfListIndex = 0;
		if(bulkData.size()>30000)
		{
			halfListIndex = bulkData.size()/2;
			
			for(int i=0;i<bulkData.size()/2;i++)
			{
				bulkRequestBody.append(actionMetaData);
			    bulkRequestBody.append(bulkData.get(i));
			    bulkRequestBody.append("\n");
			}
			String jsonString = bulkRequestBody.toString();
			bulkRequestBody.setLength(0);
			HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
			Response indexResponse = restClient.performRequest(
					"PUT",
					"/jsonindex/jsonindextype/_bulk",
					Collections.<String, String>emptyMap(),
					entity);
					
			if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
			{
				System.out.println("Index Successfully Created");
			}
		}
		
		if(bulkData.size()>30000)
		{
			for(int i = halfListIndex; i<bulkData.size();i++)
			{
				bulkRequestBody.append(actionMetaData);
				bulkRequestBody.append(bulkData.get(i));
				bulkRequestBody.append("\n");
			}
			String jsonString = bulkRequestBody.toString();
			bulkRequestBody.setLength(0);
			HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
			Response indexResponse = restClient.performRequest(
					"PUT",
					"/jsonindex/jsonindextype/_bulk",
					Collections.<String, String>emptyMap(),
					entity);
					//System.out.println(EntityUtils.toString(indexResponse.getEntity()));
			if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
			{
				System.out.println("Index Successfully Created");
			}
		}
		
		if(bulkData.size()<30000)
		{
			for (String bulkItem : bulkData) {
			    bulkRequestBody.append(actionMetaData);
			    bulkRequestBody.append(bulkItem);
			    bulkRequestBody.append("\n");
			}
			String jsonString = bulkRequestBody.toString();
			bulkRequestBody.setLength(0);
			HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
			Response indexResponse = restClient.performRequest(
					"PUT",
					"/jsonindex/jsonindextype/_bulk",
					Collections.<String, String>emptyMap(),
					entity);
					//System.out.println(EntityUtils.toString(indexResponse.getEntity()));
			if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
			{
				System.out.println("Index Successfully Created");
			}
			
		}
	 }catch(Exception e)
	{
		 System.out.println("Exception Occure");
		 e.printStackTrace();
	}

I have mentioned my jar versions used. Sometime I am not getting exception like this. Sometime it is giving exception. Please provide me solution. As per your solution I have divided my data to push into elk. Please provide solution.

6

Try with bulk with size of 1000 docs instead?

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.