The problem with getting a lot of logs

Elastic Stack Logstash
1

Hey guys. I'm new here and I apologize in advance if I opened a thread in the wrong section.
I have the task of handling a large number of logs, generating up to 100,000 per minute or even more. I currently have elk master cluster, logstash, kibana + 3 data nodes elastik configured.
There is no way I can get logstash to accept that many logs. Or maybe it's not the problem at all. So far I've achieved and my max is ~30000. The rest is all stuck in the middle of nowhere. How to find the bottleneck and what needs to be optimized ?

2

What kind of data format do you receive? Plain text, syslog, json or something else?
There are possible options:

  1. Increase the memory - config/jvm.options
  2. Use another plugin if is possible or simplify .conf
  3. Use Kafka, between source and LS
  4. Add additional LS, split source data
  5. Use a load balancer for 2 or more LS
3

Thank you for your response.

Filebeat > LS load balance > ES
image

4

the number of log lines was received more than 1 million, but in ES came in tens of times less

image
image1567×204 20.3 KB

5

Can you please explain what is your problem, too much load receiving or losing data?
How are resources utilized, CPU and memory? Have you monitor LS statics in Kibana?
Have you done LS statistic? http://localhost:9600/_node/stats/pipelines?pretty
Are you using the persistent queue?
Have you test with a small frame for instance 10 min to do stress test?

6

Hello! The data is not coming in quickly. I use persistent queue. I understand that filbeat has no time to transfer data, because logstash is constantly overloaded and has no time to process data.
Also because the log grows at an enormous speed, I use logrotate to compress and trim it. Therefore the data in this interval are lost, because they do not have time to be transferred to LS

  "pipelines" : {
    "main" : {
      "events" : {
        "in" : 37824384,
        "queue_push_duration_in_millis" : 44131783,
        "duration_in_millis" : 814999917,
        "out" : 37745165,
        "filtered" : 37745165
      },
      "flow" : {
        "input_throughput" : {
          "current" : 677.4,
          "last_1_minute" : 691.9,
          "last_5_minutes" : 729.8,
          "last_15_minutes" : 431.6,
          "last_1_hour" : 415.1,
          "lifetime" : 652.3
        },
        "output_throughput" : {
          "current" : 995.5,
          "last_1_minute" : 719.5,
          "last_5_minutes" : 670.6,
          "last_15_minutes" : 345.5,
          "last_1_hour" : 402.9,
          "lifetime" : 650.9
        },
        "queue_persisted_growth_bytes" : {
          "current" : 1293000.0,
          "last_1_minute" : 157900.0,
          "last_5_minutes" : 111700.0,
          "last_15_minutes" : 148800.0,
          "last_1_hour" : 34960.0,
          "lifetime" : 3384.0
        },
        "queue_backpressure" : {
          "current" : 0.7523,
          "last_1_minute" : 0.8275,
          "last_5_minutes" : 0.8049,
          "last_15_minutes" : 0.4745,
          "last_1_hour" : 0.4646,
          "lifetime" : 0.761
        },
        "queue_persisted_growth_events" : {
          "current" : -207.3,
          "last_1_minute" : -3.696,
          "last_5_minutes" : 0.0,
          "last_15_minutes" : 0.0,
          "last_1_hour" : 0.0,
          "lifetime" : 0.0
        },
        "filter_throughput" : {
          "current" : 995.4,
          "last_1_minute" : 719.5,
          "last_5_minutes" : 670.6,
          "last_15_minutes" : 345.5,
          "last_1_hour" : 402.9,
          "lifetime" : 650.9
        },
        "worker_concurrency" : {
          "current" : 23.48,
          "last_1_minute" : 23.52,
          "last_5_minutes" : 23.65,
          "last_15_minutes" : 11.6,
          "last_1_hour" : 10.85,
          "lifetime" : 14.05
        }
      },
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.