# Kibana with Hadoop directly? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 12, 2015, 8:41am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628 "2015-03-12T08:41:42Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![KRRK2015](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@KRRK2015](https://discuss.elastic.co/u/KRRK2015)\ **Post date:** [March 12, 2015, 8:41am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/1 "2015-03-12T08:41:42Z")
Hello, has anyone tried to get Kibana work directly with Hadoop (without elasticsearch in the middle)? If yes, how? Any references would help. Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1682c52b-a1bf-401d-81cc-a1e6cb9a41cd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1682c52b-a1bf-401d-81cc-a1e6cb9a41cd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Aaron\_Mefford](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@Aaron\_Mefford](https://discuss.elastic.co/u/Aaron_Mefford)\ **Post date:** [March 13, 2015, 1:15am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/2 "2015-03-13T01:15:19Z")
Kibana is very tightly integrated with Elasticsearch, to the point of requiring specific versions of Elasticsearch for a given version of Kibana. When you say Hadoop that really means nothing. Most of the Hadoop EcoSystem is not realtime. There are some exceptions like HBase, but their characteristics are very different from Elasticsearch. This is why it is so common to use Elasticsearch with a Hadoop environment, to provide a realtime aspect to your big data. This is why as you say Elasticsearch is in the middle. There is not reasonable way Kibana could extract random bits of data from "Hadoop". That said there are many people who have built data visualizations using Hadoop and D3, the js lib that is used in Kibana. If you really want to go without Elasticsearch (you don't), I would recommend looking for something like that. Aaron On Thursday, March 12, 2015 at 2:41:42 AM UTC-6, KRRK2015 wrote: > Hello, has anyone tried to get Kibana work directly with Hadoop (without > elasticsearch in the middle)? If yes, how? Any references would help. > Thanks. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [March 13, 2015, 3:59am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/3 "2015-03-13T03:59:09Z")
On the bright side, you can use es-hadoop connector [1] to easily get data from Hadoop/HDFS to Elasticsearch and back whatever your Hadoop stack (Map/Reduce, Cascading, Pig, Hive, Spark, Storm). [1] [Elasticsearch for Hadoop | Elastic](https://www.elastic.co/products/hadoop) On Fri, Mar 13, 2015 at 3:15 AM, [aaron@definemg.com](mailto:aaron@definemg.com) wrote: > Kibana is very tightly integrated with Elasticsearch, to the point of > requiring specific versions of Elasticsearch for a given version of Kibana. > > When you say Hadoop that really means nothing. Most of the Hadoop > EcoSystem is not realtime. There are some exceptions like HBase, but their > characteristics are very different from Elasticsearch. This is why it is > so common to use Elasticsearch with a Hadoop environment, to provide a > realtime aspect to your big data. This is why as you say Elasticsearch is > in the middle. There is not reasonable way Kibana could extract random > bits of data from "Hadoop". > > That said there are many people who have built data visualizations using > Hadoop and D3, the js lib that is used in Kibana. If you really want to go > without Elasticsearch (you don't), I would recommend looking for something > like that. > > Aaron > > On Thursday, March 12, 2015 at 2:41:42 AM UTC-6, KRRK2015 wrote: > > > Hello, has anyone tried to get Kibana work directly with Hadoop (without > > elasticsearch in the middle)? If yes, how? Any references would help. > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![KRRK2015](https://avatars.discourse-cdn.com/v4/letter/k/9d8465/32.png) [@KRRK2015](https://discuss.elastic.co/u/KRRK2015)\ **Post date:** [March 13, 2015, 9:05am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/4 "2015-03-13T09:05:03Z")
Thanks Aaron, Costin for validating what I suspected. Agreed that ES is realt-time vs. Hadoop. I was looking at Kibana as a standalone data visualization tool for batch scenarios due to its superior visualization capabilities. As you said, perhaps D3 is the only option as of now. Thanks again for your quick response. On Friday, 13 March 2015 09:29:16 UTC+5:30, Costin Leau wrote: > On the bright side, you can use es-hadoop connector [1] to easily get data > from Hadoop/HDFS to Elasticsearch and back whatever your Hadoop stack > (Map/Reduce, Cascading, Pig, Hive, Spark, Storm). > > [1] [Elasticsearch for Hadoop | Elastic](https://www.elastic.co/products/hadoop) > > On Fri, Mar 13, 2015 at 3:15 AM, \<[aa...@definemg.com](mailto:aa...@definemg.com) \\> wrote: > > > Kibana is very tightly integrated with Elasticsearch, to the point of > > requiring specific versions of Elasticsearch for a given version of Kibana. > > > > When you say Hadoop that really means nothing. Most of the Hadoop > > EcoSystem is not realtime. There are some exceptions like HBase, but their > > characteristics are very different from Elasticsearch. This is why it is > > so common to use Elasticsearch with a Hadoop environment, to provide a > > realtime aspect to your big data. This is why as you say Elasticsearch is > > in the middle. There is not reasonable way Kibana could extract random > > bits of data from "Hadoop". > > > > That said there are many people who have built data visualizations using > > Hadoop and D3, the js lib that is used in Kibana. If you really want to go > > without Elasticsearch (you don't), I would recommend looking for something > > like that. > > > > Aaron > > > > On Thursday, March 12, 2015 at 2:41:42 AM UTC-6, KRRK2015 wrote: > > > > > Hello, has anyone tried to get Kibana work directly with Hadoop (without > > > elasticsearch in the middle)? If yes, how? Any references would help. > > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c56d82c5-d16b-4a68-bf00-1777fda19811%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c56d82c5-d16b-4a68-bf00-1777fda19811%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Aaron\_Mefford](https://avatars.discourse-cdn.com/v4/letter/a/f0a364/32.png) [@Aaron\_Mefford](https://discuss.elastic.co/u/Aaron_Mefford)\ **Post date:** [March 13, 2015, 4:51pm UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/5 "2015-03-13T16:51:03Z")
To second what Costin said, it is trivial to load the data into Elasticsearch, then you have the data ready to use the full power of Kibana. Elasticsearch is very quick to setup and scales well. Trying to go the route of rolling your own setup to generate graphs with D3 will certainly be harder, and less rewarding. Just to start with you will need somewhere to store the large JSON structures you generate from Hadoop MapReduce. I do know a great JSON store, Elasticsearch. I hope you are starting to get the idea. In order to produce the web visulizations, you need to move your data from HDFS to somewhere realtime. That could be HBase or dozens of other options, but none of them will be as easy or rewarding as Elasticsearch. So why not just use Elasticsearch and Kibana? On Thu, Mar 12, 2015 at 9:59 PM, Costin Leau [costin.leau@gmail.com](mailto:costin.leau@gmail.com) wrote: > On the bright side, you can use es-hadoop connector [1] to easily get data > from Hadoop/HDFS to Elasticsearch and back whatever your Hadoop stack > (Map/Reduce, Cascading, Pig, Hive, Spark, Storm). > > [1] [Elasticsearch for Hadoop | Elastic](https://www.elastic.co/products/hadoop) > > On Fri, Mar 13, 2015 at 3:15 AM, [aaron@definemg.com](mailto:aaron@definemg.com) wrote: > > > Kibana is very tightly integrated with Elasticsearch, to the point of > > requiring specific versions of Elasticsearch for a given version of Kibana. > > > > When you say Hadoop that really means nothing. Most of the Hadoop > > EcoSystem is not realtime. There are some exceptions like HBase, but their > > characteristics are very different from Elasticsearch. This is why it is > > so common to use Elasticsearch with a Hadoop environment, to provide a > > realtime aspect to your big data. This is why as you say Elasticsearch is > > in the middle. There is not reasonable way Kibana could extract random > > bits of data from "Hadoop". > > > > That said there are many people who have built data visualizations using > > Hadoop and D3, the js lib that is used in Kibana. If you really want to go > > without Elasticsearch (you don't), I would recommend looking for something > > like that. > > > > Aaron > > > > On Thursday, March 12, 2015 at 2:41:42 AM UTC-6, KRRK2015 wrote: > > > > > Hello, has anyone tried to get Kibana work directly with Hadoop (without > > > elasticsearch in the middle)? If yes, how? Any references would help. > > > Thanks. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5a2201d3-ae86-4130-8194-0c0560223891%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/lwAEdxNsX8s/unsubscribe](https://groups.google.com/d/topic/elasticsearch/lwAEdxNsX8s/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAJogdmca%3DbH2mPep5j3WUdjz8YT3UvPSuGG0UvFvjPMZvWwzHA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAF9vEErEb0H%2BfAbDVZLrHhUhr\_1LwH7YF5BF0RE-S8\_XTX3Bwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAF9vEErEb0H%2BfAbDVZLrHhUhr_1LwH7YF5BF0RE-S8_XTX3Bwg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:26am UTC](https://discuss.elastic.co/t/kibana-with-hadoop-directly/22628/6 "2017-07-06T00:26:58Z")