# Logstash fell under load

**URL:** <https://discuss.elastic.co/t/logstash-fell-under-load/269766>\
**Category:** Logstash\
**Created:** [April 10, 2021, 9:07am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766 "2021-04-10T09:07:29Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 10, 2021, 9:07am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/1 "2021-04-10T09:07:29Z")

</div>

Hello. I have three clients. They send logs via `filebeat` to main server with `logstash`. A few days ago there are a lot of logs somewhere around 12 M in an amount. The `logstash` couldn't process all this logs and it fell. Does it have any throttling settings to avoid this situation again?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 10, 2021, 4:20pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/2 "2021-04-10T16:20:38Z")

</div>

You need to share the pipeline configuration you are using and your `logstash.yml` to help understand what is happening.

What do you run on the logstash machine, only logstash or other applications? What is the hardware specs? Are you using persisted queues or in-memory queues?

Also, share the logstash log with the errors when it crashed.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 10, 2021, 4:48pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/3 "2021-04-10T16:48:41Z")

</div>

Hi @Andrey_RF

In addition to the above.

did you make any changes / scale up the JVM heap for Logstash see [here](https://www.elastic.co/guide/en/logstash/current/jvm-settings.html) ..

If you are on larger host you can certainly ignore the 8GB top, I have had to scale above that for some intensive workloads.

A not to point out the obvious, there are a couple good sections in the docs on [scaling](https://www.elastic.co/guide/en/logstash/current/deploying-and-scaling.html) and [performance tuning](https://www.elastic.co/guide/en/logstash/current/performance-tuning.html).

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 11, 2021, 11:54am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/4 "2021-04-11T11:54:18Z")

</div>

`pipeline`:

```auto
input {
  beats {
    port => 5000
  }
}

filter {

  if 'django' in [tags] {
    grok {
      break_on_match => false
      match => {
        "message" => [
          "^%{LOGLEVEL:log-level} %{TIMESTAMP_ISO8601:timestamp}",
          "Пользователь - (?<login>[^\;]+)",
          "Имя: (?<name>[^\;]+)",
          "id - (?<user-id>[^\;]+)",
          "email - (?<email>[^\;]+)",
          "ip - %{IP:ip}",
          'Запрос: "(?<request>[^\"]+)',
          'Метод: "(?<method>[^\"]+)',
          "Модуль: (?<module>[^\;]+)",
          "Функция: (?<func>[^\;]+)"
        ]
      }
    }
  } else if 'gunicorn' in [tags] {
    grok {
      match => { "message" => '^%{LOGLEVEL:log-level} %{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:message}' }
      overwrite => ["message"]
    }

     grok {
      break_on_match => false
      match => { "message" => [
        '^ \n\tСообщение: %{IP:ip} "%{WORD:http-method} (?<http-url>[^\s]+) (?<http-protocol>[^\"]+)" (?<status>[\d]+) "URL: (?<url>[^\"]+)" "[^\"]+";',
        'В модуле: (?<module>[^\n]+)'
        ]
      }
    }
  } else if 'nginx' in [tags] {
    grok {
      match => {
         "message" => [
           '^%{IP:client-ip} - - \[(?<timestamp>[\d]+/[\w]+/[\d]+:[\d]+:[\d]+:[\d]+) \+[\d]+\] "%{WORD:http-method} (?<http-url>[^\s]+) (?<http-protocol>[^\"]+)" (?<status-code>[\d]+) (?<bytessent>[\d]+) "(?<refferer>[^\"]+)" "(?<user-agent>[^\"]+)" "-"$',
           '^%{IP:client-ip} - - \[(?<timestamp>[\d]+/[\w]+/[\d]+:[\d]+:[\d]+:[\d]+) \+[\d]+\]',
           "^(?<timestamp>[\d]+/[\d]+/[\d]+ [\d]+:[\d]+:[\d]+) \[%{LOGLEVEL:log-level}\]"
         ]
      }
    }
  } else {
    grok {
      match => [
        "message", "(?m)%{TIMESTAMP_ISO8601:timestamp} \[%{LOGLEVEL:log-level}\]%{GREEDYDATA:message}", # rabbitmq.info

        "message", "\[%{TIMESTAMP_ISO8601:timestamp}: %{LOGLEVEL:log-level}", # celery.log

        "message", "%{TIMESTAMP_ISO8601:timestamp}" # попытка просто достать лог
      ]
    }
  }

  date {
      match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.SSS", "yyyy/MM/dd HH:mm:ss", "dd/MMM/yyyy:HH:mm:ss", "ISO8601"]
      timezone => "Europe/Moscow"
      remove_field => ["timestamp"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logstash-%{[host][hostname]}"
  }
}

```

`filebeat.yml`:

```auto
# ============================== Filebeat inputs ===============================

filebeat.inputs:

- type: log
  enabled: true

  paths:
    - /var/log/eias-web/backend.*.log

  tags: ["django"]

  multiline.type: pattern
  multiline.pattern: '^INFO|^ERROR|^WARNING|^CRITICAL'
  multiline.negate: true
  multiline.match: after

- type: log
  enable: true

  paths:
    - /var/log/eias-web/fingerprint.info.log
    - /var/log/eias-web/gunicorn.*.log

  tags: ["gunicorn", "fingerpirnt"]

  multiline.type: pattern
  multiline.pattern: '^INFO|^ERROR|^WARNING|^CRITICAL'
  multiline.negate: true
  multiline.match: after

- type: log
  enabled: true

  paths:
    - /var/log/eias-web/celery.log

  tags: ["celery"]
  miltiline.type: pattern
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after

- type: log
  enable: true

  paths:
    - /var/log/eias-web/crash.*
    - /var/log/eias-web/rabbitmq.*.log

  tags: ["crash", "rabbitmq"]

  multiline.type: pattern
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after

# filestream is an experimental input. It is going to replace log input in the future.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  
# ============================== Filebeat registry =============================

filebeat.registry.path: ${path.data}/registry
filebeat.registry.file_permissions: 0600

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# ================================= Dashboards =================================

# =================================== Kibana ===================================

# =============================== Elastic Cloud ================================

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ip:5000"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    # - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

```

It runs `logstash`, `elasticsearch` and `kibana` on the same machine. It's 192 G free space. Idk 🙂 How can I check it ?

`logstash` is in `docker` and logs don't save.

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 11, 2021, 11:56am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/5 "2021-04-11T11:56:39Z")

</div>

It's `docker-compose` settings for `logstash`:

```auto
logstash:
    build:
      context: logstash/
    container_name: logstash
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "5000:5000"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 11, 2021, 2:00pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/6 "2021-04-11T14:00:13Z")

</div>

First thing I would try is set Logstash heap to 4GB.

You could mount the Logstash logs to the a volume so you could see the logs.

How much heap are you giving elasticsearch?

How much total RAM and CPU on the server this is all running on?

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 11, 2021, 9:56pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/7 "2021-04-11T21:56:25Z")

</div>

> [@stephenb](#):
>
> ch heap are you giving el

```auto
  elasticsearch:
    build:
      context: elasticsearch/
    container_name: elasticsearch
    volumes:
      - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
      - /home/elasticsearch:/usr/share/elasticsearch/data
    ports:
      - "9200:9200"
    environment:
      ES_JAVA_OPTS: "-Xmx4096m -Xms4096m"

```

```auto
free -h
              total used free shared buff/cache available
Mem: 11G 5.5G 317M 69M 5.7G 5.7G
Swap: 5.9G 39M 5.8G

```

```auto
cat /proc/cpuinfo | grep core
cpu cores	: 2
cpu MHz : 1995.000

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 11, 2021, 10:12pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/8 "2021-04-11T22:12:37Z")

</div>

Ok

So 4GB to elasticsearch  
So try to give 2GB to Logstash.

Not a huge server to run this all on...

Only 2 cores.. that elasticsearch and Logstash are fighting to over.

It is Generally not best practice to run Logstash and elasticsearch on same server.

With docker so/so ...

Maybe for small testing but for production you would need a bigger server... And make sure elasticsearch and Logstash have plenty of ram and CPU

The 11M logs over what time frame?

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 11, 2021, 11:29pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/9 "2021-04-11T23:29:03Z")

</div>

We have not so big a website now. It's only 200-400 requests per hour and `ELK` works good. We will extend the machine if the load increase. I just want to find something like throttling to avoid the situation like this.

It was 11 M logs for 15-30 minutes.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 12, 2021, 12:00am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/10 "2021-04-12T00:00:51Z")

</div>

Well that is quite a spike right

10M events in 30 min is 5.5K events / sec that is several thousand times your normal load.... It's going to be very hard to design a system that is both only for 400 per hour but also 5000 per second those are two pretty different systems.

BUT that said you could look at the persistent queue ... But I still not sure that'll work.

If you know when that spike is going to come you could scale up log stash and then scale it back down.

But this is a classic system design question of designing for average or peak usage... There is always trade-offs.

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 12, 2021, 1:09am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/11 "2021-04-12T01:09:09Z")

</div>

I see. It was at first time for all time. Mb it was a ddos or something else.

I will read about queue in logstash, thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 12, 2021, 1:15am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/12 "2021-04-12T01:15:34Z")

</div>

Yeah usually people _try_ to detect DDOS closer to the edge /FW etc or you could do something like Kafka which is much better at managing back pressure but go ahead and try the persistent queue first but that is a huge Spike

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 12, 2021, 1:25am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/13 "2021-04-12T01:25:59Z")

</div>

Ohhh ... apologies @Andrey_RF

I maybe misunderstood was this the very first time you started logstash? if so it may be going back and reading all the old logs / files in the directory... that's what it does so it may try to have loaded all the old logs...

Or was it definitely a spike within time?

---

<div class="post-metadata">

**Author:** ![Andrey\_RF](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_rf/32/86484_2.png) [@Andrey\_RF](https://discuss.elastic.co/u/Andrey_RF)\
**Post date:** [April 12, 2021, 8:56am UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/14 "2021-04-12T08:56:15Z")

</div>

No no no. It had worked a week before it fallen. I have a correct log's timestamp so you can see a chart.

 ![Screenshot_20210412_115555](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7717ae75492d26eef15a181d0bb3ae8cdaf0abe.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 12, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/15 "2021-04-12T16:54:31Z")

</div>

Yup that wont be easy... 🙂

You could try putting in more heap and some persistent queues... that may or may not help.

While the system keeps up... there will be nothing in the queue.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-fell-under-load/269766/16 "2021-05-10T16:54:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.