Unable to compile spark 2.0 project with es-hadoop

Elastic Stack Elasticsearch
1

I have the following dependency added in sbt

"org.elasticsearch" %% "elasticsearch-spark-20" % "5.5.2"

I get the following error

[warn] Run 'evicted' to see detailed eviction warnings
[info] Compiling 1 protobuf files to /opt/aruba/central/apps/mon-ae-notifications/alert_detector/target/src_managed/main
[info] Compiling schema /opt/aruba/central/apps/mon-ae-notifications/alert_detector/src/main/protobuf/notifications.proto
protoc-jar: protoc version: 320, detected platform: linux/amd64
protoc-jar: executing: [/tmp/protocjar2523904227258434471/bin/protoc.exe, --plugin=protoc-gen-scala=/tmp/protocbridge1934192420034201130, --scala_out=grpc:/opt/aruba/central/apps/mon-ae-notifications/alert_detector/target/src_managed/main, -I/opt/aruba/central/apps/mon-ae-notifications/alert_detector/src/main/protobuf, -I/opt/aruba/central/apps/mon-ae-notifications/alert_detector/target/protobuf_external, /opt/aruba/central/apps/mon-ae-notifications/alert_detector/src/main/protobuf/notifications.proto]
[info] Compiling protobuf
[info] Protoc target directory: /opt/aruba/central/apps/mon-ae-notifications/alert_detector/target/src_managed/main
[info] Compiling 15 Scala sources to /opt/aruba/central/apps/mon-ae-notifications/alert_detector/target/classes...
[error] missing or invalid dependency detected while loading class file 'StreamingContext.class'.
[error] Could not access term hadoop in package org.apache,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
[error] A full rebuild may help if 'StreamingContext.class' was compiled against an incompatible version of org.apache.
[error] missing or invalid dependency detected while loading class file 'StreamingContext.class'.
[error] Could not access type Configuration in value org.apache.conf,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.)
[error] A full rebuild may help if 'StreamingContext.class' was compiled against an incompatible version of org.apache.conf.
[error] two errors found
[error] (AlertDetector/compile:compileIncremental) Compilation failed
[error] Total time: 67 s, completed 16 Aug, 2017 9:42:15 PM

2

This may be the issue. We exclude Hadoop from our dependencies. You should try adding an explicit dependency on Hadoop.

3

You are right, I fixed this by including hadoop dependencies but I forgot to close this thread.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.