System design for monitoring more than 1250 nodes


(ChiBi PonD) #1

Does anyone can help me design the suitable system for monitoring 1250 nodes at 600kHz? I have no idea how many elasticsearch master nodes I need. Or should I really have to use Logstash for input instead of doing direct curl to the master node. Thank you in advance.


(Magnus B├Ąck) #2

You keep asking the same question and you've received a couple of answers, including advice to drop the idea of using curl commands to insert data.

If you're going to get anywhere near 600k insertions/s you're going to need a bunch of nodes for the indexing alone, but if you're going to store data for any time at all you're going to need a huge cluster anyway. 600k/s is almost 52 billion documents per day. To give you some kind of idea, I have a three-node cluster for two billion documents.

The Performance Considerations for Elasticsearch Indexing blog post from September 2014 contains a number of considerations. More recently there was also a blog post about how to estimate the size of clusters. I suggest you start there.


(Mark Walkom) #3

Yep, please use one of your existing threads @ChiBiPonD :slight_smile:


(Mark Walkom) #4