Does anyone can help me design the suitable system for monitoring 1250 nodes at 600kHz? I have no idea how many elasticsearch master nodes I need. Or should I really have to use Logstash for input instead of doing direct curl to the master node. Thank you in advance.
You keep asking the same question and you've received a couple of answers, including advice to drop the idea of using curl commands to insert data.
If you're going to get anywhere near 600k insertions/s you're going to need a bunch of nodes for the indexing alone, but if you're going to store data for any time at all you're going to need a huge cluster anyway. 600k/s is almost 52 billion documents per day. To give you some kind of idea, I have a three-node cluster for two billion documents.
The Performance Considerations for Elasticsearch Indexing blog post from September 2014 contains a number of considerations. More recently there was also a blog post about how to estimate the size of clusters. I suggest you start there.
1 Like
Yep, please use one of your existing threads @ChiBiPonD 