Connection Error while writing

Hi,

During ES Write operations getting Connection Error, that seems to be because of huge data size being transferred from one node to other over network. Please suggest how to limit data packet size. PSB more details:

2019-03-27 23:39:48,377 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Number of splits :4
Total Length = 268333437
Input split[0]:
Length = 173137245
Locations:


Input split[1]:
Length = 80474020
Locations:


Input split[2]:
Length = 9510775
Locations:


Input split[3]:
Length = 5211397
Locations:


'es.nodes = HOST',
	'es.index.auto.create = false',
'es.net.http.auth.user =USER', 
'es.net.http.auth.pass = PASS',
'es.net.ssl = false',
'es.net.ssl.keystore.location =  KEY_STORE_PATH', 
'es.net.ssl.keystore.pass = KEY_STORE_PASS',
'es.net.ssl.truststore.location =TRUST_STORE_PATH',
'es.net.ssl.truststore.pass = TRUST_STORE_PASS',
'es.net.ssl.cert.allow.self.signed = true',
'es.http.retries=5', 
'es.mapping.id = XXXX',
'es.batch.write.retry.count=2000', 
'es.batch.size.entries=1000');

SET mapreduce.input.fileinputformat.split.minsize 268435456;
SET pig.maxCombinedSplitSize 268435456;
SET pig.splitCombination true;

Exception:
org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed;
at org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:142)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:423)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:403)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:407)
at org.elasticsearch.hadoop.rest.RestClient.refresh(RestClient.java:277)
at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:276)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.doClose(EsOutputFormat.java:214)
at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.close(EsOutputFormat.java:196)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.close(PigOutputFormat.java:149)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.