# ELK and Hadoop integration
**URL:** https://discuss.elastic.co/t/elk-and-hadoop-integration/28286
**Category:** Elasticsearch
**Tags:** es-hadoop
**Created:** [August 29, 2015, 2:54pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286 "2015-08-29T14:54:29Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@ajays](https://discuss.elastic.co/u/ajays)
#### Post date: [August 29, 2015, 2:54pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/1 "2015-08-29T14:54:29Z")
Hi,
We are having one ELK setup and logs from various sources like apache, nginx, system messages etc are processed. We keep the ES data for 1 month and dump the rest to s3 bucket. For analysing the history data we need to again restore the content of s3 to elasticsearch. Now I want to integrate ELK with hadoop so that we can see the old data in kibana. I heard of elasticsearch-hadoop. Can anybody plz help how to dump all data to hadoop via elasticsearch and vice versa?
LogStash---\>ElasticSearch---\>Hadoop
AjayS
---
### Author:  [@ajays](https://discuss.elastic.co/u/ajays)
#### Post date: [August 29, 2015, 6:54pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/2 "2015-08-29T18:54:56Z")
Hi,
I saw on web that we can use hive/pig to get the data from hadoop to elasticsearch but what are the ways to do other way round. Can we use elasticsearch-hadoop connectors for this purpose? Can somebody give me some example/link?
Regards,
AjayS
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [August 29, 2015, 7:16pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/3 "2015-08-29T19:16:28Z")
Have you looked at the project [reference documentation](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/reference.html) ? It explains the architecture, configuration and how to both read and write data through the connector for each library with code examples.
---
### Author:  [@ajays](https://discuss.elastic.co/u/ajays)
#### Post date: [August 30, 2015, 2:16pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/4 "2015-08-30T14:16:36Z")
@costin,
I watched your webinar video and I am trying your example but getting this error.
[root@localhost bin]# ./hive -hiveconf hive.aux.jars.path=///opt/elasticsearch-hadoop-2.1.1/dist/elasticsearch-hadoop-2.1.1.jar
Logging initialized using configuration in jar:file:/usr/local/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
hive\> CREATE EXTERNAL TABLE eslogs (time STRING, extension STRING, clientip STRING, request STRING, response INT, agent STRING) STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler' TBLPROPERTIES('es.nodes' = '192.168.44.175','es.port' = '9200','es.resource' = 'demo/hive','es.mapping.names' = 'time:@timestamp');
FAILED: SemanticException Cannot find class 'org.elasticsearch.hadoop.hive.ESStorageHandler'
Regards,
AjayS
---
### Author:  [@costin](https://discuss.elastic.co/u/costin)
#### Post date: [August 31, 2015, 8:55pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/5 "2015-08-31T20:55:08Z")
The classpath is not properly set; use an hdfs classpath instead of a local one since Hive tends to be picky about these things:
[https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#\_installation\_3](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#_installation_3)
Your distro documentation might/should provide more info
---
### Author:  [@ajays](https://discuss.elastic.co/u/ajays)
#### Post date: [September 1, 2015, 3:26am UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/6 "2015-09-01T03:26:12Z")
@costin,
Thanks. Let me check and get back to you with the results.
Regards,
AjayS
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:27pm UTC](https://discuss.elastic.co/t/elk-and-hadoop-integration/28286/7 "2017-07-06T13:27:36Z")