Elasticsearch/logstash/kibana with hadoop as Hunk alternative

Elastic Stack Elasticsearch
1

Hi

For performance improvement I'm trying to combine
Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource
alternative to Hunk. Unfortunately I'm familiar only with HDFS where I
store logs. In my opinion the combination of Elasticsearch and Hadoop
should use HDFS as storage and transparent Hadoop map/reduce functionality
for search.

I ran through elasticsearch-hadoop documentation and unfortunately I didn't
understand how this combination could help me for Kibana log analysis.
Documentation says "Elasticsearch real-time search and analytics natively
integrated with Hadoop.". But what should I configure? Hadoop with
Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found
only Java code parts, nothing about the Hadoop configuration, so it seems
that I should be familiar with Java programming. As for the last one I
found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was
developed for indexes backup/restore, am I right?

Anyway, are my expectations right? Or elasticsearch-hadoop was developed
for Hadoop developers only and it is not suitable for
"elasticsearch/logstash/kibana + hadoop" (like Hunk).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cb3a8c8e-a279-422b-8967-c9e2a544ca9e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I've found another alternative. New 3.6 version of HUE supports
data visualization in kibana style:
http://gethue.com/hadoop-ui-hue-3-6-and-the-search-dashboards-are-out/

четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал:

Hi

For performance improvement I'm trying to combine
Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource
alternative to Hunk. Unfortunately I'm familiar only with HDFS where I
store logs. In my opinion the combination of Elasticsearch and Hadoop
should use HDFS as storage and transparent Hadoop map/reduce functionality
for search.

I ran through elasticsearch-hadoop documentation and unfortunately I
didn't understand how this combination could help me for Kibana log
analysis. Documentation says "Elasticsearch real-time search and analytics
natively integrated with Hadoop.". But what should I configure? Hadoop with
Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found
only Java code parts, nothing about the Hadoop configuration, so it seems
that I should be familiar with Java programming. As for the last one I
found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was
developed for indexes backup/restore, am I right?

Anyway, are my expectations right? Or elasticsearch-hadoop was developed
for Hadoop developers only and it is not suitable for
"elasticsearch/logstash/kibana + hadoop" (like Hunk).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/22278760-02e8-4c68-80eb-cc3a316ea85d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Unfortunately Hue uses python script for data transformation. So it is not
correct to compare logstash/elasticsearch/kibana with new "Search" in Hue
3.6.

Anyway the initial question is still open.

четверг, 19 июня 2014 г., 18:55:55 UTC+4 пользователь kay rus написал:

I've found another alternative. New 3.6 version of HUE supports
data visualization in kibana style:
Hue - The open source SQL Assistant for Data Warehouses

четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал:

Hi

For performance improvement I'm trying to combine
Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource
alternative to Hunk. Unfortunately I'm familiar only with HDFS where I
store logs. In my opinion the combination of Elasticsearch and Hadoop
should use HDFS as storage and transparent Hadoop map/reduce functionality
for search.

I ran through elasticsearch-hadoop documentation and unfortunately I
didn't understand how this combination could help me for Kibana log
analysis. Documentation says "Elasticsearch real-time search and analytics
natively integrated with Hadoop.". But what should I configure? Hadoop with
Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found
only Java code parts, nothing about the Hadoop configuration, so it seems
that I should be familiar with Java programming. As for the last one I
found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was
developed for indexes backup/restore, am I right?

Anyway, are my expectations right? Or elasticsearch-hadoop was developed
for Hadoop developers only and it is not suitable for
"elasticsearch/logstash/kibana + hadoop" (like Hunk).

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi,

The Elasticsearch Hadoop integration acts as a connector between Hadoop
(one cluster) to Elasticsearch (another cluster). The two systems do not
run one of top of the other but rather one service interacts with the
other. So within your Hadoop cluster one would run a job that reads/writes
data to Elasticsearch; and with es-hadoop this becomes either (than
without) since ES looks and acts as if it's an HDFS resource.

Hope this clarifies things a bit. There's a section in the docs that
focuses on this topic which you can find here [1]

[1]

On Mon, Jun 23, 2014 at 11:26 AM, kay rus kay.diam@gmail.com wrote:

Unfortunately Hue uses python script for data transformation. So it is not
correct to compare logstash/elasticsearch/kibana with new "Search" in Hue
3.6.

Anyway the initial question is still open.

четверг, 19 июня 2014 г., 18:55:55 UTC+4 пользователь kay rus написал:

I've found another alternative. New 3.6 version of HUE supports
data visualization in kibana style:
Hue - The open source SQL Assistant for Data Warehouses

четверг, 19 июня 2014 г., 10:42:50 UTC+4 пользователь kay rus написал:

Hi

For performance improvement I'm trying to combine
Elasticsearch/Logstash/Kibana with Hadoop (cdh4) and configure opensource
alternative to Hunk. Unfortunately I'm familiar only with HDFS where I
store logs. In my opinion the combination of Elasticsearch and Hadoop
should use HDFS as storage and transparent Hadoop map/reduce functionality
for search.

I ran through elasticsearch-hadoop documentation and unfortunately I
didn't understand how this combination could help me for Kibana log
analysis. Documentation says "Elasticsearch real-time search and analytics
natively integrated with Hadoop.". But what should I configure? Hadoop with
Elasticsearch or Elasticsearch with Hadoop? As for the first one, I found
only Java code parts, nothing about the Hadoop configuration, so it seems
that I should be familiar with Java programming. As for the last one I
found only "Hadoop HDFS Snapshot/Restore plugin", but I guess it was
developed for indexes backup/restore, am I right?

Anyway, are my expectations right? Or elasticsearch-hadoop was developed
for Hadoop developers only and it is not suitable for
"elasticsearch/logstash/kibana + hadoop" (like Hunk).

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/24713fe4-abd7-40b2-a8a4-67d7b60182e3%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmfR4z36TUDmWRyu5HhG8%2BuO2dqr%2BoVTT8CB4gdPSSj1Ew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.