Doubts about hardware requirements for elasticsearch

Elastic Stack Elasticsearch
1

I'm trying to setup elasticsearch cluster. I will get maximum of 4GB of data every day.

So what will be hardware required to set up ElasticSearch 7.6.2 and kibana 7.6.2

What is ideal configuration for server- side RAM/Hard/disks/CPU.

I currently have a single node cluster with:

  • Intel (R) Xeon (R) CPU E7- 4807 @ 1.87GHz CPU (s): 4
  • 20GB RAM
  • 1TB HDD.

As additional data, the logs will be collected by Decanter 2.0.0.

Appreciate your help!!

2

Welcome to our community! :smiley:

If you are only using a single node cluster, that should be fine. But please use 7.7, it's the latest release :slight_smile:

3

Thank you very much for the welcome

With the previous configuration it would be enough to index 4 GB daily without having any type of problem.

How many documents per second should be indexed.

Thank you so much.

4

What do you mean?

5

How can I know how many documents per second my cluster is capable of indexing.

6

Ah ok, you will need to test that yourself as it's dependant on the hardware, use case etc :slight_smile:

7

For example I tried indexing 31000 documents in a java test that pulls in parallel with decanter logs to elasticsearch and I see that when finishing the elasticsearch test it continues indexing like 20 minutes later and it doesn't index me almost in real time as I expected, and I don't know what it is the problem, is it due to disk speed or do I have to have more hadware.

8

That's not how it works. As soon as your client has received a 200 response from Elasticsearch, then it's been indexed. What's happening after that might be merging, but it's hard to know without more information on what you are seeing.

9

My single node cluster has a shard with no replica, refresh_interval: 30s was defined and
each document is approximately 1 kb in size.

10

The first thing to check is that you have followed these instructions, especially about using bulk requests (especially since you are using HDD storage). Elasticsearch is often limited by disk utilization as indexing and associated merging can be very I/O intensive.

11

Thank you very much Christian_Dahlqvist.
I'm going to review those instructions.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.