ELK and Hadoop integration

Elastic Stack Elasticsearch
1

Hi,

We are having one ELK setup and logs from various sources like apache, nginx, system messages etc are processed. We keep the ES data for 1 month and dump the rest to s3 bucket. For analysing the history data we need to again restore the content of s3 to elasticsearch. Now I want to integrate ELK with hadoop so that we can see the old data in kibana. I heard of elasticsearch-hadoop. Can anybody plz help how to dump all data to hadoop via elasticsearch and vice versa?

LogStash--->ElasticSearch--->Hadoop

AjayS

2

Hi,

I saw on web that we can use hive/pig to get the data from hadoop to elasticsearch but what are the ways to do other way round. Can we use elasticsearch-hadoop connectors for this purpose? Can somebody give me some example/link?

Regards,
AjayS

3

Have you looked at the project reference documentation ? It explains the architecture, configuration and how to both read and write data through the connector for each library with code examples.

4

@costin,

I watched your webinar video and I am trying your example but getting this error.

[root@localhost bin]# ./hive -hiveconf hive.aux.jars.path=///opt/elasticsearch-hadoop-2.1.1/dist/elasticsearch-hadoop-2.1.1.jar

Logging initialized using configuration in jar:file:/usr/local/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive> CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip STRING, request STRING, response INT, agent STRING) STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler' TBLPROPERTIES('es.nodes' = '192.168.44.175','es.port' = '9200','es.resource' = 'demo/hive','es.mapping.names' = 'time:@timestamp');
FAILED: SemanticException Cannot find class 'org.elasticsearch.hadoop.hive.ESStorageHandler'

Regards,
AjayS

5

The classpath is not properly set; use an hdfs classpath instead of a local one since Hive tends to be picky about these things:
https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#_installation_3

Your distro documentation might/should provide more info

6

@costin,

Thanks. Let me check and get back to you with the results.

Regards,
AjayS