Latency issue

Elastic Community and Ecosystem
1

Hi everyone,

I'm an IT student, and i work for a company. My internship is to develop a solution to log, analyze and visualize the event logs.

I do not have many financial resources for this project. So I set up a POC on 3 servers of the ELK stack .

The hardware configurations of my servers are as follows:

  • 2 CPU, Intel(R) Xeon 2,27GHz (4 cores)
  • 24 Go RAM
  • 1,5 To SAS disk

Each server has this configuration.

My ELK stack is composed of Redis, Logstash, ElasticSearch and Kibana.

Here is my architecture:

Pasted image1062×829 11.5 KB

Logstash heap size is 1G
Elasticsearch Client heap size is 6G
Elasticsearch Master heap size is 2G
Elasticsearch Data heap size is 4G

I have 5 shard, 1 replica.

Today I get the logs of a dozen servers. In 4 months I have 3 billion ( 2,05To ) documents in my cluster.

I plan to retrieve the logs to hundred servers.

In recent days I have observed a slowdown of performance in my cluster. Kibana crash often, the informations takes a long time to appear, etc.

I wanted to know how I can tell if the data load is too large for my cluster.

What tests should I perform ?

thanks for the help :grinning: