How to control/limit ELK disk usage

[ManuelF (Manuel)]

Hi,

Running ELK 7.11.2 Standalone on Debian 10

My ELK machine has 500 GB of disk space. This is the second time the machine runs out of disk space (100% usage). When this happens, I am not able to log into Kibana, as Elasticsearch needs to write to index sessions2-*, but since there's not enough space, login becomes impossible. When this happens, I have to use tool Elasticsearch-Curator and set actions to remove indexes, to make some room and then be able to login. Once I'm in, I noticed that I have severalwinlogbeat-* indexes of 50 GB each. The ELK node is currently monitoring 23 servers with Winlogbeat and Endpoint Security agents running on each machine.

In an attempt to deal with this issue, I have:

1. Set curator to run every day and remove some indexes older than 7 days, where I have included winlogbeat-* and some other indexes that may grow large.
2. Set an events filter for each Winlogbeat agent to ship only the event's that I need.

Please see below the actions.yml for curator and the events filter for Winlogbeat.

Please advise if there is anything wrong or missing with my configs or if there is any other mechanism that I could use to control/limit the ELK disk usage.

actions.yml

actions:
  1:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  2:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: auditbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  3:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  4:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: heartbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  5:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: packetbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  6:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: winlogbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  7:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: metricbeat-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7
  8:
    action: delete_indices
    description: "Delete indices older than 7 days (based on index creation date)"
    options:
      disable_action: False
      ignore_empty_list: True
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: prefix
      value: sessions2-
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 7

Winlogbeat events filter

  - name: Security
    processors:
      - drop_event.when.not.or:
          - equals.winlog.event_id: 104
          - equals.winlog.event_id: 1022
          - equals.winlog.event_id: 1033
          - equals.winlog.event_id: 1034
          - equals.winlog.event_id: 1102
          - equals.winlog.event_id: 4100
          - equals.winlog.event_id: 4616
          - equals.winlog.event_id: 4624
          - equals.winlog.event_id: 4625
          - equals.winlog.event_id: 4634
          - equals.winlog.event_id: 4648
          - equals.winlog.event_id: 4656
          - equals.winlog.event_id: 4657
          - equals.winlog.event_id: 4663
          - equals.winlog.event_id: 4672
          - equals.winlog.event_id: 4688
          - equals.winlog.event_id: 4689
          - equals.winlog.event_id: 4697
          - equals.winlog.event_id: 4698
          - equals.winlog.event_id: 4699
          - equals.winlog.event_id: 4700
          - equals.winlog.event_id: 4701
          - equals.winlog.event_id: 4702
          - equals.winlog.event_id: 4719
          - equals.winlog.event_id: 4720
          - equals.winlog.event_id: 4722
          - equals.winlog.event_id: 4723
          - equals.winlog.event_id: 4725
          - equals.winlog.event_id: 4728
          - equals.winlog.event_id: 4732
          - equals.winlog.event_id: 4735
          - equals.winlog.event_id: 4737
          - equals.winlog.event_id: 4738
          - equals.winlog.event_id: 4740
          - equals.winlog.event_id: 4755
          - equals.winlog.event_id: 4756
          - equals.winlog.event_id: 4767
          - equals.winlog.event_id: 4769
          - equals.winlog.event_id: 4772
          - equals.winlog.event_id: 4777
          - equals.winlog.event_id: 4782
          - equals.winlog.event_id: 4946
          - equals.winlog.event_id: 4947
          - equals.winlog.event_id: 4950
          - equals.winlog.event_id: 4954
          - equals.winlog.event_id: 4964
          - equals.winlog.event_id: 5025
          - equals.winlog.event_id: 5031
          - equals.winlog.event_id: 5140
          - equals.winlog.event_id: 6281
          - equals.winlog.event_id: 7035
          - equals.winlog.event_id: 7036
          - equals.winlog.event_id: 7040
          - equals.winlog.event_id: 7045
          - equals.winlog.event_id: 40961