# Elasticsearch/logstash/kibana with hadoop as Hunk alternative **URL:** \ **Category:** Elasticsearch\ **Created:** [June 19, 2014, 6:42am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202 "2014-06-19T06:42:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![kay\_kay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_kay/32/1470_2.png) [@kay\_kay](https://discuss.elastic.co/u/kay_kay)\ **Post date:** [June 19, 2014, 6:42am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202/1 "2014-06-19T06:42:50Z")
Hi For performance improvement I'm trying to combine Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource alternative to Hunk. Unfortunately I'm familiar only with HDFS where I store logs. In my opinion the combination of Elasticsearch and Hadoop should use HDFS as storage and transparent Hadoop map/reduce functionality for search. I ran through elasticsearch-hadoop documentation and unfortunately I didn't understand how this combination could help me for Kibana log analysis. Documentation says "Elasticsearch real-time search and analytics natively integrated with Hadoop.". But what should I configure? Hadoop with Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found only Java code parts, nothing about the Hadoop configuration, so it seems that I should be familiar with Java programming. As for the last one I found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was developed for indexes backup/restore, am I right? Anyway, are my expectations right? Or elasticsearch-hadoop was developed for Hadoop developers only and it is not suitable for "elasticsearch/logstash/kibana + hadoop" (like Hunk). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cb3a8c8e-a279-422b-8967-c9e2a544ca9e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cb3a8c8e-a279-422b-8967-c9e2a544ca9e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![kay\_kay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_kay/32/1470_2.png) [@kay\_kay](https://discuss.elastic.co/u/kay_kay)\ **Post date:** [June 19, 2014, 2:55pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202/2 "2014-06-19T14:55:55Z")
I've found another alternative. New 3.6 version of HUE supports data visualization in kibana style: [http://gethue.com/hadoop-ui-hue-3-6-and-the-search-dashboards-are-out/](http://gethue.com/hadoop-ui-hue-3-6-and-the-search-dashboards-are-out/) четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал: > Hi > > For performance improvement I'm trying to combine > Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource > alternative to Hunk. Unfortunately I'm familiar only with HDFS where I > store logs. In my opinion the combination of Elasticsearch and Hadoop > should use HDFS as storage and transparent Hadoop map/reduce functionality > for search. > > I ran through elasticsearch-hadoop documentation and unfortunately I > didn't understand how this combination could help me for Kibana log > analysis. Documentation says "Elasticsearch real-time search and analytics > natively integrated with Hadoop.". But what should I configure? Hadoop with > Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found > only Java code parts, nothing about the Hadoop configuration, so it seems > that I should be familiar with Java programming. As for the last one I > found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was > developed for indexes backup/restore, am I right? > > Anyway, are my expectations right? Or elasticsearch-hadoop was developed > for Hadoop developers only and it is not suitable for > "elasticsearch/logstash/kibana + hadoop" (like Hunk). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/22278760-02e8-4c68-80eb-cc3a316ea85d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/22278760-02e8-4c68-80eb-cc3a316ea85d%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![kay\_kay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kay_kay/32/1470_2.png) [@kay\_kay](https://discuss.elastic.co/u/kay_kay)\ **Post date:** [June 23, 2014, 8:26am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202/3 "2014-06-23T08:26:42Z")
Unfortunately Hue uses python script for data transformation. So it is not correct to compare logstash/elasticsearch/kibana with new "Search" in Hue 3.6. Anyway the initial question is still open. четверг, 19 июня 2014 г., 18:55:55 UTC+4 пользователь kay rus написал: > I've found another alternative. New 3.6 version of HUE supports > data visualization in kibana style: > [Hue - The open source SQL Assistant for Data Warehouses](http://gethue.com/hadoop-ui-hue-3-6-and-the-search-dashboards-are-out/) > > четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал: > > > Hi > > > > For performance improvement I'm trying to combine > > Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource > > alternative to Hunk. Unfortunately I'm familiar only with HDFS where I > > store logs. In my opinion the combination of Elasticsearch and Hadoop > > should use HDFS as storage and transparent Hadoop map/reduce functionality > > for search. > > > > I ran through elasticsearch-hadoop documentation and unfortunately I > > didn't understand how this combination could help me for Kibana log > > analysis. Documentation says "Elasticsearch real-time search and analytics > > natively integrated with Hadoop.". But what should I configure? Hadoop with > > Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found > > only Java code parts, nothing about the Hadoop configuration, so it seems > > that I should be familiar with Java programming. As for the last one I > > found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was > > developed for indexes backup/restore, am I right? > > > > Anyway, are my expectations right? Or elasticsearch-hadoop was developed > > for Hadoop developers only and it is not suitable for > > "elasticsearch/logstash/kibana + hadoop" (like Hunk). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [June 23, 2014, 1:52pm UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202/4 "2014-06-23T13:52:41Z")
Hi, The Elasticsearch Hadoop integration acts as a connector between Hadoop (one cluster) to Elasticsearch (another cluster). The two systems do not run one of top of the other but rather one service interacts with the other. So within your Hadoop cluster one would run a job that reads/writes data to Elasticsearch; and with es-hadoop this becomes either (than without) since ES looks and acts as if it's an HDFS resource. Hope this clarifies things a bit. There's a section in the docs that focuses on this topic which you can find here [1] [1] > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Mon, Jun 23, 2014 at 11:26 AM, kay rus [kay.diam@gmail.com](mailto:kay.diam@gmail.com) wrote: > Unfortunately Hue uses python script for data transformation. So it is not > correct to compare logstash/elasticsearch/kibana with new "Search" in Hue > 3.6. > > Anyway the initial question is still open. > > четверг, 19 июня 2014 г., 18:55:55 UTC+4 пользователь kay rus написал: > > > I've found another alternative. New 3.6 version of HUE supports > > data visualization in kibana style: > > [Hue - The open source SQL Assistant for Data Warehouses](http://gethue.com/hadoop-ui-hue-3-6-and-the-search-dashboards-are-out/) > > > > четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал: > > > > > Hi > > > > > > For performance improvement I'm trying to combine > > > Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource > > > alternative to Hunk. Unfortunately I'm familiar only with HDFS where I > > > store logs. In my opinion the combination of Elasticsearch and Hadoop > > > should use HDFS as storage and transparent Hadoop map/reduce functionality > > > for search. > > > > > > I ran through elasticsearch-hadoop documentation and unfortunately I > > > didn't understand how this combination could help me for Kibana log > > > analysis. Documentation says "Elasticsearch real-time search and analytics > > > natively integrated with Hadoop.". But what should I configure? Hadoop with > > > Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found > > > only Java code parts, nothing about the Hadoop configuration, so it seems > > > that I should be familiar with Java programming. As for the last one I > > > found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was > > > developed for indexes backup/restore, am I right? > > > > > > Anyway, are my expectations right? Or elasticsearch-hadoop was developed > > > for Hadoop developers only and it is not suitable for > > > "elasticsearch/logstash/kibana + hadoop" (like Hunk). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmfR4z36TUDmWRyu5HhG8%2BuO2dqr%2BoVTT8CB4gdPSSj1Ew%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmfR4z36TUDmWRyu5HhG8%2BuO2dqr%2BoVTT8CB4gdPSSj1Ew%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:20am UTC](https://discuss.elastic.co/t/elasticsearch-logstash-kibana-with-hadoop-as-hunk-alternative/18202/5 "2017-07-06T01:20:35Z")