I am now pushing half of the data in one request and half of the data in second request. Though I am getting the below exception : listener timeout after waiting for [30000] ms
Below is my code :
try {
JSONObject obj = new JSONObject();
FileInputStream file = new FileInputStream(new File("C:\\Users\\cajitb\\Desktop\\TempFiles\\op\\TestExcel01.xlsx"));
Workbook workbook = new Workbook(file);
Worksheet worksheet = workbook.getWorksheets().get(0);
Cells cells = worksheet.getCells();
Range range = cells.getMaxDisplayRange();
int totalColumns = range.getColumnCount();
int totalRows = range.getRowCount();
RowCollection rows = cells.getRows();
int maxColumn = cells.getMaxDataColumn();
int maxRows = cells.getMaxDataRow();
SimpleDateFormat format = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
Date d1 = new Date();
System.out.println("Start Time:"+format.format(d1));
List<Object> colList = new ArrayList<Object>();
String actionMetaData = String.format("{ \"index\" : { \"_index\" : \"%s\", \"_type\" : \"%s\" } }%n", "jsonindex", "jsonindextype");
for(int i = 0; i<=0 ; i++)
{
for (int j = 0; j < totalColumns; j++)
{
colList.add(cells.get(i, j).getValue());
}
}
List<String> bulkData = new ArrayList<String>();
for (int i = 1; i < rows.getCount(); i++)
{
for (int j = 0; j < totalColumns; j++)
{
String key = (String)colList.get(j);
if(null==key || "".equals(key))
key = "Column"+j;
obj.put(key, cells.get(i, j).getValue());
}
Date d = new Date();
obj.put("Action_Date", dateFormat.format(d));
String jsonString = obj.toJSONString();
bulkData.add(jsonString);
obj.clear();
}
StringBuilder bulkRequestBody = new StringBuilder();
int halfListIndex = 0;
if(bulkData.size()>30000)
{
halfListIndex = bulkData.size()/2;
for(int i=0;i<bulkData.size()/2;i++)
{
bulkRequestBody.append(actionMetaData);
bulkRequestBody.append(bulkData.get(i));
bulkRequestBody.append("\n");
}
String jsonString = bulkRequestBody.toString();
bulkRequestBody.setLength(0);
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
Response indexResponse = restClient.performRequest(
"PUT",
"/jsonindex/jsonindextype/_bulk",
Collections.<String, String>emptyMap(),
entity);
if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
{
System.out.println("Index Successfully Created");
}
}
if(bulkData.size()>30000)
{
for(int i = halfListIndex; i<bulkData.size();i++)
{
bulkRequestBody.append(actionMetaData);
bulkRequestBody.append(bulkData.get(i));
bulkRequestBody.append("\n");
}
String jsonString = bulkRequestBody.toString();
bulkRequestBody.setLength(0);
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
Response indexResponse = restClient.performRequest(
"PUT",
"/jsonindex/jsonindextype/_bulk",
Collections.<String, String>emptyMap(),
entity);
//System.out.println(EntityUtils.toString(indexResponse.getEntity()));
if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
{
System.out.println("Index Successfully Created");
}
}
if(bulkData.size()<30000)
{
for (String bulkItem : bulkData) {
bulkRequestBody.append(actionMetaData);
bulkRequestBody.append(bulkItem);
bulkRequestBody.append("\n");
}
String jsonString = bulkRequestBody.toString();
bulkRequestBody.setLength(0);
HttpEntity entity = new NStringEntity(jsonString, ContentType.APPLICATION_JSON);
Response indexResponse = restClient.performRequest(
"PUT",
"/jsonindex/jsonindextype/_bulk",
Collections.<String, String>emptyMap(),
entity);
//System.out.println(EntityUtils.toString(indexResponse.getEntity()));
if(indexResponse.getStatusLine().getStatusCode() == HttpStatus.SC_OK)
{
System.out.println("Index Successfully Created");
}
}
}catch(Exception e)
{
System.out.println("Exception Occure");
e.printStackTrace();
}
I have mentioned my jar versions used. Sometime I am not getting exception like this. Sometime it is giving exception. Please provide me solution. As per your solution I have divided my data to push into elk. Please provide solution.