How to control/limit ELK disk usage

ManuelF · June 2, 2021, 1:20pm

Hi,

Running ELK 7.11.2 Standalone on Debian 10

My ELK machine has 500 GB of disk space. This is the second time the machine runs out of disk space (100% usage). When this happens, I am not able to log into Kibana, as Elasticsearch needs to write to index sessions2-*, but since there's not enough space, login becomes impossible. When this happens, I have to use tool Elasticsearch-Curator and set actions to remove indexes, to make some room and then be able to login. Once I'm in, I noticed that I have severalwinlogbeat-* indexes of 50 GB each. The ELK node is currently monitoring 23 servers with Winlogbeat and Endpoint Security agents running on each machine.

In an attempt to deal with this issue, I have:

Set curator to run every day and remove some indexes older than 7 days, where I have included winlogbeat-* and some other indexes that may grow large.
Set an events filter for each Winlogbeat agent to ship only the event's that I need.

Please see below the actions.yml for curator and the events filter for Winlogbeat.

Please advise if there is anything wrong or missing with my configs or if there is any other mechanism that I could use to control/limit the ELK disk usage.

actions.yml

actions:
  1:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  2:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: auditbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  3:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  4:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: heartbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  5:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: packetbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  6:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: winlogbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  7:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: metricbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  8:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: sessions2-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

Winlogbeat events filter

  - name: Security
    processors:
      - drop_event.when.not.or:
          - equals.winlog.event_id: 104
          - equals.winlog.event_id: 1022
          - equals.winlog.event_id: 1033
          - equals.winlog.event_id: 1034
          - equals.winlog.event_id: 1102
          - equals.winlog.event_id: 4100
          - equals.winlog.event_id: 4616
          - equals.winlog.event_id: 4624
          - equals.winlog.event_id: 4625
          - equals.winlog.event_id: 4634
          - equals.winlog.event_id: 4648
          - equals.winlog.event_id: 4656
          - equals.winlog.event_id: 4657
          - equals.winlog.event_id: 4663
          - equals.winlog.event_id: 4672
          - equals.winlog.event_id: 4688
          - equals.winlog.event_id: 4689
          - equals.winlog.event_id: 4697
          - equals.winlog.event_id: 4698
          - equals.winlog.event_id: 4699
          - equals.winlog.event_id: 4700
          - equals.winlog.event_id: 4701
          - equals.winlog.event_id: 4702
          - equals.winlog.event_id: 4719
          - equals.winlog.event_id: 4720
          - equals.winlog.event_id: 4722
          - equals.winlog.event_id: 4723
          - equals.winlog.event_id: 4725
          - equals.winlog.event_id: 4728
          - equals.winlog.event_id: 4732
          - equals.winlog.event_id: 4735
          - equals.winlog.event_id: 4737
          - equals.winlog.event_id: 4738
          - equals.winlog.event_id: 4740
          - equals.winlog.event_id: 4755
          - equals.winlog.event_id: 4756
          - equals.winlog.event_id: 4767
          - equals.winlog.event_id: 4769
          - equals.winlog.event_id: 4772
          - equals.winlog.event_id: 4777
          - equals.winlog.event_id: 4782
          - equals.winlog.event_id: 4946
          - equals.winlog.event_id: 4947
          - equals.winlog.event_id: 4950
          - equals.winlog.event_id: 4954
          - equals.winlog.event_id: 4964
          - equals.winlog.event_id: 5025
          - equals.winlog.event_id: 5031
          - equals.winlog.event_id: 5140
          - equals.winlog.event_id: 6281
          - equals.winlog.event_id: 7035
          - equals.winlog.event_id: 7036
          - equals.winlog.event_id: 7040
          - equals.winlog.event_id: 7045
          - equals.winlog.event_id: 40961

system · June 30, 2021, 1:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Free up disk space Logstash curator	2	1693	September 17, 2020
How to delete older logs in ELK to give each application a certain disk quota? Elasticsearch	7	3109	July 5, 2017
Curator: When the disk space used by all indices exceeds a certain limit Kibana curator	2	356	August 3, 2022
How to manage elasticsearch logs: disk almost full Elasticsearch	14	2370	November 30, 2020
How to limit amount of incoming data Elasticsearch	3	745	January 12, 2017

How to control/limit ELK disk usage

actions.yml

Winlogbeat events filter

Related topics