Configuration for High Volume Search 10k/sec


(William Pearson) #1

Cluster Id: b3ceb5

I am currently running elastic search in Elastic cloud. I have a single node cluster with 64 gb of ram and 1.5 terabytes of hard drive space. I am doing high volume of searches. Users upload csv files with the search terms, I use Nest to generate lucene queries and submit them in parallel to elastic search server. I noticed at one point I maxed the cpu out at 10k request/second and response time of the node increased significantly, 80 seconds to return a document. What type of configuration would I need to overcome this issue? Is the single node my point of failure?

Thanks in Advance.


(Mark Walkom) #2

Yes, a single node is a bad idea because you have no redundancy or load balancing, it'd be a good start.


(William Pearson) #3

Thank you for your response. I am looking at moving to the following configuration: 1 Client Node, 1 Master, 2 Data Nodes. My server configuration for each of the data nodes would be 64 gb of ram, 2 TB Hard Drive with 8 cores. The data nodes would also be marked master eligible. My single master node and single client node would each have 8gb ram, 90g hard drive space and 4 cores. I am hoping the configuration will solve my load issues. Does the specifications mentioned seem out of whack?


(Mark Walkom) #4

Elastic Cloud doesn't yet have dedicated node types.
Are you suggesting moving to a standalone cluster?


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.