We upgraded from ES2.3.4 to ES5.1.2 and when creating the transport client using the following code
val settings: Settings = Settings.builder()
.put("transport.ping_schedule", "5s")
.put("request.headers.X-Found-Cluster", "${cluster.name}")
//.put("shield.transport.ssl", "true")
// .put("action.bulk.compress","false")
// .put("shield.ssl.ciphers","TLS_RSA_WITH_AES_128_CBC_SHA256")
// .put("shield.user", esCredentials)
.put("cluster.name", esClusterName).build()
val esclient = new PreBuiltTransportClient(settings)
esclient.addTransportAddress(new InetSocketTransportAddress(InetAddress.getByName(esEndPoint), esPort.toInt))'
Compilation is fine but getting runtimeexception as
java.lang.NoSuchMethodError: io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio /netty/buffer/CompositeByteBuf; (org.apache.spark.deploy.yarn.ApplicationMaster)
java.lang.NoSuchMethodError: io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;
at org.elasticsearch.transport.netty4.Netty4Utils.toByteBuf(Netty4Utils.java:78)
at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:449)
at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:91)
at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:976)
at org.elasticsearch.transport.TcpTransport.sendRequest(TcpTransport.java:958)
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:520)
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:465)
at org.elasticsearch.transport.TransportService.submitRequest(TransportService.java:451)
at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.doSample(TransportClientNodesService.java:403)
at org.elasticsearch.client.transport.TransportClientNodesService$NodeSampler.sample(TransportClientNodesService.java:338)
at org.elasticsearch.client.transport.TransportClientNodesService.addTransportAddresses(TransportClientNodesService.java:179)
at org.elasticsearch.client.transport.TransportClient.addTransportAddress(TransportClient.java:301)
at utility.EsConnector$.getConnection(EsConnector.scala:53)
and my build.sbt is as follows
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
"org.apache.spark" %% "spark-streaming-kafka" % sparkVersion,
"org.apache.spark" %% "spark-sql" % sparkVersion % "provided",
"com.google.guava" % "guava" % "18.0",
("org.elasticsearch" %% "elasticsearch-spark-13" % "5.1.2").
exclude("org.apache.hadoop", "hadoop-yarn-api").
exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
exclude("org.eclipse.jetty.orbit", "javax.servlet").
exclude("org.slf4j", "slf4j-api") intransitive(),
("org.elasticsearch" % "elasticsearch-hadoop" % "5.0.0-alpha4").
exclude("org.apache.hadoop", "hadoop-yarn-api").
exclude("org.eclipse.jetty.orbit", "javax.mail.glassfish").
exclude("org.eclipse.jetty.orbit", "javax.servlet").
exclude("org.slf4j", "slf4j-api") intransitive(),
//excludeAll ExclusionRule(organization = "org.apache.hadoop"),
"org.elasticsearch" % "elasticsearch" % "5.1.2",
"org.elasticsearch.client" % "transport" % "5.1.2",
//"org.elasticsearch.plugin" % "transport-netty4-client" % "5.1.2",
"org.apache.logging.log4j" % "log4j-core" % "2.7",
"org.apache.logging.log4j" % "log4j-api" % "2.7",
// "io.netty" % "netty-buffer" % "4.1.8.Final",
"joda-time" % "joda-time" % "2.7",
"com.databricks" %% "spark-xml" % "0.4.1",
"com.databricks" %% "spark-csv" % "1.4.0",
"com.sun.jersey" % "jersey-servlet" % "1.19",
"com.typesafe" % "config" % "1.3.0",
"org.springframework.boot" % "spring-boot-starter-security" % "1.3.3.RELEASE",
"com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4" % "provided",
"com.fasterxml.jackson.core" % "jackson-core" % "2.4.4" % "provided",
"com.nimbusds" % "nimbus-jose-jwt" % "4.26.1",
"com.spotify" % "dns" % "3.1.1",
"com.amazonaws" % "aws-java-sdk-s3" % "1.11.76",
"org.apache.httpcomponents" % "httpclient" % "4.3.6" % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-common" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-hdfs" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-app" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-jobclient" % "2.7.3" % "provided",
"org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.7.3" % "provided",
"org.elasticsearch.plugin" % "shield" % "2.3.4" from "https://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/2.3.4/shield-2.3.4.jar",
s"spectre-$service_stage-gen" % "spectre-dls-baseservice" % s"$baseservice_ver" from s"http://engci-maven.cisco.com/artifactory/spectre-$service_stage-gen-group/data-lake/spectre-dls-baseservice/$baseservice_ver/spectre-dls-baseservice-$baseservice_ver.jar",
s"spectre-$service_stage-gen" % "spectre-base-java" % s"$microservicebase_ver" from s"http://engci-maven-master.cisco.com/artifactory/spectre-$service_stage-gen/com/cisco/spectre/microservice/spectre-base-java/$microservicebase_ver/spectre-base-java-$microservicebase_ver.jar"
)
resolvers ++= Seq(
"Akka Repository" at "http://repo.akka.io/releases/",
"scala-tools" at "https://oss.sonatype.org/content/groups/scala-tools"
//"elasticsearch-releases" at "https://maven.elasticsearch.org/releases"
)
Is this a known issue?