Elasticsearch audit log take more size

Hi Team,

Elasticsearch audit log take 20-25 GB size every day. I'm using elasticsearch 7.16.2 and same version for filebeat. I have enable the audit key by enabling the below keys in elasticsearch.yml file.

xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.emit_request_body: true
xpack.security.audit.logfile.emit_node_name: true
xpack.security.audit.logfile.events.include: ["authentication_success"]

Below I have added filebeat.yml file configuration.

###################### Filebeat Configuration #########################

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   - D:\Elastic\logs\elasticsearch_audit-*.json

Can you help me to minimize the size.

Hi Team,

Elastic search audit log take more space. What I can do for this so that audit log file size will reduce? Please help me.

Thanks,
Nitin Bisht

This setting is very verbose... Which is probably adding significantly to the size of your logs

Do you really need that?

Hi @stephenb ,

I'm going to set the mention key as false. And I will let you know what is the result after getting one day log.

xpack.security.audit.logfile.events.emit_request_body: false

Thanks,
Nitin Bisht

@Nitin08bisht This will definitely have a big effect. This setting will log the full request body of every write and read, which results in more then double the amount of data then your ingesting. When I enable this, my Elastic cluster explodes lol.

Unfortunately this setting enables extensive auditing, which sometimes is needed for compliance etc. If only Elastic would give us the ability to enable this auditing setting only for some high sensitive indices.

Willem

Hi @stephenb /@willemdh

Thanks for the solution. It really help me a lot. The size of elasticsearch auditlogs has reduced considerably as compared to earlier.

Before applying the below keys in elasticsearch.yml file the elasticsearch auditlogs file take too much size. But when I applied the below key, then it reduced the size of elasticsearch auditlogs file.

xpack.security.audit.logfile.events.emit_request_body: false

Thanks,
Nitin Bisht

Hi @willemdh

Yes agreed Elastic needs to provide better/ easier audit logging.

Yup totally true if you gave _all event types.

One thing you can do, is in the filebeat audit module add drop events that you are not interested does not contain the indices you are interested... not great but can be done.. there or in a/the ingest pipeline

2 Likes

@stephenb Thanks for the suggestion, but imho that's not an ideal solution for multiple reasons, mostly related to unnecessary load (big envs). Atm I'm filtering my audit logs in a Logstash pipeline, because the available filtering options in Filebeat are not granular enough tbh.

2 Likes

Totally agree... Not ideal... Logstash was not mentioned, that is good place to do the filtering... Better will be when elasticsearch provides better native granularity.

1 Like