I am overseeing a production cluster consisting of approximately 83 nodes, and we are handling nearly a million of data points daily . To optimize the production environment, what are the recommended best practices regarding hardware specifications? Specifically, how many data nodes should be employed, how much RAM should each machine have, and what other hardware specifications should be considered for optimal performance?
This will depend a lot on the use case, data volumes, type of data and the type of load and latencies the cluster must serve. As you have a current production cluster it is probably good to start by looking at how this is performing and identifying any potential performance issues or where there may be excessive capacity.
You will need to provide a lot more information about the use case, including data, query and indexing volumes for someone to be able to give any recommendations.
Information about the setup and configuration of the current cluster and how it is performing would also be useful. A good way to provide an overview of the cluster is to provide the full output of the cluster stats API.
It is not clear what you mean by this. Indexing one million documents per day in a 83 node cluster sounds like very light indexing load.
++ for instance this benchmark involves a 3-node cluster with some configurations indexing about 200,000 documents per second. 1M/day seems very low even for a single node, let alone 83 nodes!
@Christian_Dahlqvist I’m currently facing data loss at the ingest node, where some data packets are being lost. Additionally, my data nodes are not being utilized evenly. I ingest a total of 1 TB of data each week, starting with a small amount that gradually increases to 1 TB, and this data is deleted after one week. @DavidTurner
If you want someone to be able to help you will need to provide a lot more details than that about the current cluster, data volumes and indexing and query load. The full output of the API I linked to would be a good start.
i just want suggest or solution that why i am lossing data on ingest node .. is it related to hardware defficency ? do i need to increase SSD of system or do i need to use coordinating node before ingest node soo that may balance evenly and i dont face data loss
Given that you are not willing to provide any information I would recommend you go through the guide about tuning for indexing speed.This webinar might also be useful.