Elastic Cluster with Hadoop

[shikhasingh]

Hi Team,

I have created one node ES cluster and want to integrate with my hadoop cluster, we are using ES 6.6 and HDP 3 version. we want to integrate ES with hive and followed the given documents.
But when try to create an external table after connecting to hive on same node where ES cluster is running it says "Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only' (state=08S01,code=1)"

Command which I ran is CREATE EXTERNAL TABLE elastic_table(name string, ID int, profession string) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'customer','es.nodes.wan.only' = 'true','es.index.auto.create'='TRUE');

I suspect this is due to our hadoop cluster is kerberised, could you please guide us to make ES cluster accessible to our hadoop cluster.
Looking for response.

Thanks a lot,
Shikha

[shikhasingh]

I have found https://docs.search-guard.com/latest/search-guard-installation#first-time-installation-full-cluster-restart and trying to install searchguard so that i can configure the kerberos for ES.

Have downloaded 24.1 version of SearchGuard for ES 6.6.1 from mentioned link --

Security for Elasticsearch | Latest Search Guard Versions | Search Guard

When try to run " ./elasticsearch-plugin install -b /etc/elasticsearch-6.6.1/search-guard-6-6.6.1-24.1.zip" but getting below error :-

ERROR: Unknown plugin /etc/elasticsearch-6.6.1/search-guard-6-6.6.1-24.1.zip

Kindly suggest.

Thanks,
Shikha

[ikakavas (Ioannis Kakavas)]

Hi there,

Kerberos is available given the right license in the Elastic stack and our documentation is here.

Searchguard is a 3rd party plugin, and you decide to use that you will probably get help quicker opening an issue at their relevant github repository

[shikhasingh]

Hi,

Thanks for the reply, We are using open source free version of ES, I have gone through many links and found that Security feature is not available for free version.

Please correct me if i am wrong.
Our Hadoop cluster is kerberised and if we need to integrate ES with it we need to kerberised our ES cluster as well.

Please let me know if we should go with only licensed version of ES.

Thanks,
Shikha