Hi,
I'm new to Elasticsearch. I have set up 4 machine with Elasticsearch installed. One is the master and the rest are data nodes. And I have written a bash script to run the XPOST curl concurrently and the posting outcome is very poor.
I want to use Elasticsearch to store a large amount of values ~600KHz (or 600,000 values per second). Is it possible to do this? Please help me how to achieve it. Thank you in advance.
Your going to want more nodes it does not have to be 1 node per machine and at least 2 more masters. (or let them all be master/data nodes)
This may help you get started in tuning
curl would be very slow as you have startup time for the CLI so you may also want to use Logstash to consume and insert your data instead of curl.
I would also suggest in install Marvel so you can see what the performance of your indexing inside is.
Heap Size can affect your indexing too.
Suppose that I have 3 machines and set them to be master/data nodes. What is the next thing that I have to do? I have no idea how to use Logstash to consume some value that I want (Do I need to develop some other app to get my desire value) and insert them to Elasticsearch. I have tested with curl sending value directly and it is very slow as you told.
Thank you so much for helping me and your suggestion. Mr.eperry