ELK stack running on Marathon

Elastic Stack Elasticsearch
1

Good evening everyone,

As part of a new project within my company we decided to use ELK stack.

One of the pre-requisites of such project is that all components shall have HA configured.

For ElasticSearch HA comes by default, but for Logstash and Kibana we didn't realize and easy way to make it (yes we can use Linux OS Cluster approach), but project see that approach as no flexible

And alternative is being considered and consists on configure a Mesos environment (3 master, 3 slaves) with marathon running on top. (we've just deployed such environment using latest versions available for mesos and marathon)

Our plan is to deploy 3 instances of ElasticSearch, 1 of Logstash and 1 of Kibana, each of them as a docker containers

At least logstash and kibana servers must be accessible by systems outside the mesos cluster, which as far as my understand means we need to set a mesos-dns system with a configuration in our global dns in order to foward each request to mesos cluster to our mesos-dns server

I've reading multiple information about docker containers over marathon, but to be honest I've not a clear clue of howto configure elasticsearch,logstash and kibana docker containers to run on top of marathon over mesos cluster

If anyone can give any tip or point me to some document that can help me to configure the ELK docker containers to run on top of marathon will be great

As well any info about mesos-dns will be much welcome.

Thanks in advance for your help and support

Kind Regards

2

As it happens I just wrote a few paragraphs about Logstash and "high availability":

For Elasticsearch HA comes by default, but for Logstash and Kibana we didn't realize and easy way to make it (yes we can use Linux OS Cluster approach), but project see that approach as no flexible

Kibana persists all state in Elasticsearch so you can just run it on multiple machines behind a load balancer. Kibana 4 basically just acts as a proxy to Elasticsearch so it's very lean on resources.

Most of your other questions are probably more Marathon questions than they are ELK questions. Logstash and Kibana can be viewed as pretty generic TCP (and possibly UDP) based services.

3

Hi Magnus,

Thanks for flash answer, it helped me to reorganize part of the scenario

Working with marathon community I've fixed some of the open topics

Kind Regards