There's no easy solution in this case since there's no 'constant'. Basically the environment, in this case YARN, doesn't
offer any facilities or guarantees for long-lived processes. Pushing that towards the clients (the user, Kibana,
Logstash) is not a solution since these care about the service (ES) not about YARN. What you need is some type of
binding process that allows one to tie a known address/port within YARN to a service known to it so, in case of a move,
the binding still remains in place; however I'm not aware of such a solution.
There is work underway in YARN for long-lived processes so potentially this can be addressed.
As a workaround one could create a stand-alone client node that can be part of the same cluster and use that to connect
to ES - basically having a known-location to which ES can be accessed.
P.S. Note this is not an ES specific problem; it affects any listening/server-like software.
On 2/5/15 5:14 PM, Douglas Moore wrote:
With YARN, the ES nodes can move from server to server upon restarts.
How do Logstash and Kibana discover the right server IP & port for the ES listener(s) while ES is running under YARN?
What do those config fragments look like. We could not find anything online about that.
Thanks in advance.
On Tuesday, February 3, 2015 at 4:10:50 PM UTC-5, Costin Leau wrote:
Hi,
Whether ES is running on YARN, Linux, Windows, Docker or AWS doesn't matter to the clients as long as they have access
to the instance.
In other words, logstash doesn't see any difference in ES if it's running on Linux vs YARN.
However one has to take into account the difference in the environment when it comes to configuring the storage for ES,
opening up the firewall, etc...
When running ES on YARN you have to double check first whether the server has started correctly and whether that
port is
not closed by the firewall.
If that's not the case, consider using a different port (by changing the configuration and redeploying ES). There might
be other settings in YARN that can
affect the behavior outside ES itself making the connection impossible.
Hope this helps,
On 2/3/15 10:13 PM, A_Aziz wrote:
> Hi,
>
> I am using Hortonworks Sandbox 2.1 running on a VM. My goal is to run ELK within the VM using ES as a YARN application.
>
> I have followed the instructions
> on:http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/ey-install.html
<http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/2.1.Beta/ey-install.html>. I can see ES running within the
> containers.
>
> I also got logstash running and tested using some of the tutorials. I also tested it with ES running on linux as a
> stand-alone and it worked perfectly.
>
> My question is how do I get logstash to connect/write to ES-YARN? I am using the default config and I cannot access ES
> using curl on port 9200.
>
> Also, is the only way to interact(read/write) with ES-YARN through a MapReduce, Pig, Hive, or other "Hadoop" applications?
>
> If there is a guide/tutorial on how to work with logstash and ES-YARN, please forward it to me.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
>elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/8133f505-c33a-4581-8ec6-1343eeef7904%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/8133f505-c33a-4581-8ec6-1343eeef7904%40googlegroups.com>
> <https://groups.google.com/d/msgid/elasticsearch/8133f505-c33a-4581-8ec6-1343eeef7904%40googlegroups.com?utm_medium=email&utm_source=footer
<https://groups.google.com/d/msgid/elasticsearch/8133f505-c33a-4581-8ec6-1343eeef7904%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2be460c9-088d-4a2e-a440-647d09000896%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2be460c9-088d-4a2e-a440-647d09000896%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/54D39018.4000102%40gmail.com.
For more options, visit https://groups.google.com/d/optout.