Monitoring an Elastic Search cluster

Elastic Stack Elasticsearch
1

Hello Elastic experts,

Any existing built-in tools to monitor a local setup Elastic Search cluster? Also, if the tool could send alert if cluster is not stable/has issues, it will be great.

thanks in advance,
Lin

2

suggest use marvel, it very powerful, or you can use zabbix integrate with elasticsearch restful api. because marvel not support alerts

3

Thank you Junheng_Gong.

regards,
Lin

4

I used graphite with the elasticsearch graphite plugin and grafana to view
the data. Marvel is perhaps better, but it is a paid product.

Cheers,

Ivan

5

We use ELK to monitor Elasticsearch. The Head plugin is also very useful.

-- Asaf.

6

Thanks Asaf,

Does it support alert?

regards,
Lin

7

Lin,

We have implemented Alerts on top of ELK and we offer this as a SaaS solution - You can implement on your own or you can use other services that offer this.

I want also elaborate why we use ELK for monitoring Elasticsearch - From what we found we realized that poor performance is usually linked to logical problem with ways people ingest logs and it was crucial for us to combine the metrics with the logs in order to gain this visibility. We found for example that mapping exceptions have a significant impact on performance and therefor we make sure to Alert on any time we face a mapping issue.

Our set up is like this:

  1. We use Docker to wrap collectl and sent data to ELK - This is a Docker container that we have developed and you are welcome to use.
  2. We use ELK to collect logs from all the machine starting from kernel logs and up to application level logs and this is how we found issues like cluster disconnects.

All of this can be achieved with the open source version of ELK. We also have some guidelines on how to deploy ELK for production use.

Hope that helps.

-- Asaf.

8

Thanks Asaf,

What do you mean "mapping exceptions have a significant impact on performance"? Appreciate if you could show us an example?

regards,
Lin

9

This is dependent on your use case but if, for example, you see frequent exceptions like "org.elasticsearch.index.mapper.MapperParsingException: failed to parse" in your log files it has sever implications on the health of the cluster.

You can also read about issues we had with the cluster here - (http://logz.io/blog/elasticsearch-cluster-disconnects/)

There are plenty other issues in the logs that should be monitored and if you good coverage you need to be able to monitor the logs as well as the health of the cluster.

-- Asaf.

10

Thanks Asaf, for mapping, you mean ElasticSearch index document into cluster?

regards,
Lin

11

I mean index documents into Elasticsearch

-- Asaf.

12

Thanks Asaf.