I am currently running elastic search in Elastic cloud. I have a single node cluster with 64 gb of ram and 1.5 terabytes of hard drive space. I am doing high volume of searches. Users upload csv files with the search terms, I use Nest to generate lucene queries and submit them in parallel to elastic search server. I noticed at one point I maxed the cpu out at 10k request/second and response time of the node increased significantly, 80 seconds to return a document. What type of configuration would I need to overcome this issue? Is the single node my point of failure?
Thank you for your response. I am looking at moving to the following configuration: 1 Client Node, 1 Master, 2 Data Nodes. My server configuration for each of the data nodes would be 64 gb of ram, 2 TB Hard Drive with 8 cores. The data nodes would also be marked master eligible. My single master node and single client node would each have 8gb ram, 90g hard drive space and 4 cores. I am hoping the configuration will solve my load issues. Does the specifications mentioned seem out of whack?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.