Help for extra large load

Hi Everybody,
I have 20 machines that each one has about 100,000 logs per minute, I'm using elk right now but the problem is sometimes logs are not showing real-time in kibana discovery. I used tuning, performance got better but the delayed logs were not fixed. any suggestions or tips fellas?

I think you need to provide some additional information in order to allow someone to help, e.g:

  • Which version of Elasticsearch are you using?
  • What is the full output of the cluster stats API?
  • What is the specification of your cluster with respect to node types, configuration and hardware used (CPUs, RAM and type of storage used)?
  • Have you followed these guidelines?
  • How are you indexing data into Elasticsearch?
  • How many different types of data are you ingesting? How many different indices and shards are you actively indexing into?
1 Like

Hi dear Christian
I'm using version 7.16
I have 9 Elasticsearch and 5 logstash
8 core CPU and 16 G ram on each elastic, 8 core CPU and 12 G ram on each logstash.
yes, I used these guidelines.
3 kinds of log levels (custom logs) about 5 indices 1 shard .
this problem appears sometimes not always.
I can't find the problem I don't know if it's a network problem or something else.
thank you for your time friend

do you have any idea what should I do?

Can you please answer all the questions I asked?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.