Elasticsearch audit log take more size

Elastic Stack Elasticsearch
1

Hi Team,

Elasticsearch audit log take 20-25 GB size every day. I'm using elasticsearch 7.16.2 and same version for filebeat. I have enable the audit key by enabling the below keys in elasticsearch.yml file.

xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.emit_request_body: true
xpack.security.audit.logfile.emit_node_name: true
xpack.security.audit.logfile.events.include: ["authentication_success"]

Below I have added filebeat.yml file configuration.

###################### Filebeat Configuration #########################

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   - D:\Elastic\logs\elasticsearch_audit-*.json

Can you help me to minimize the size.

2

Hi Team,

Elastic search audit log take more space. What I can do for this so that audit log file size will reduce? Please help me.

Thanks,
Nitin Bisht

3

This setting is very verbose... Which is probably adding significantly to the size of your logs

Do you really need that?

4

Hi @stephenb ,

I'm going to set the mention key as false. And I will let you know what is the result after getting one day log.

xpack.security.audit.logfile.events.emit_request_body: false

Thanks,
Nitin Bisht

5

@Nitin08bisht This will definitely have a big effect. This setting will log the full request body of every write and read, which results in more then double the amount of data then your ingesting. When I enable this, my Elastic cluster explodes lol.

Unfortunately this setting enables extensive auditing, which sometimes is needed for compliance etc. If only Elastic would give us the ability to enable this auditing setting only for some high sensitive indices.

Willem

6

Hi @stephenb /@willemdh

Thanks for the solution. It really help me a lot. The size of elasticsearch auditlogs has reduced considerably as compared to earlier.

Before applying the below keys in elasticsearch.yml file the elasticsearch auditlogs file take too much size. But when I applied the below key, then it reduced the size of elasticsearch auditlogs file.

xpack.security.audit.logfile.events.emit_request_body: false

Thanks,
Nitin Bisht

7

Hi @willemdh

Yes agreed Elastic needs to provide better/ easier audit logging.

Yup totally true if you gave _all event types.

One thing you can do, is in the filebeat audit module add drop events that you are not interested does not contain the indices you are interested... not great but can be done.. there or in a/the ingest pipeline

2 Likes
8

@stephenb Thanks for the suggestion, but imho that's not an ideal solution for multiple reasons, mostly related to unnecessary load (big envs). Atm I'm filtering my audit logs in a Logstash pipeline, because the available filtering options in Filebeat are not granular enough tbh.

2 Likes
9

Totally agree... Not ideal... Logstash was not mentioned, that is good place to do the filtering... Better will be when elasticsearch provides better native granularity.

1 Like
10

Hi @stephenb /@willemdh ,

Audit logs are still taking 20gb size per day after set mentioned key as false. Requesting you to please look into this issue as me and Nitin are in the same team and facing the same issue.

Regards,
Syed

11

Have you looked at the actual logs and confirmed only the events you want are being logged.

Second @syed0510 this is a community forum not paid support.

I am confused this seems to indicate the logs reduced considerably.

I suggest you look at the audit settings.

Other things you could does is set the codec to best compress and / or drop some of the repeated host / agent fields with a processor in auditbeat.

There is no magic you have control... In general Elasticsearch auditing is pretty verbose...

12

Hi @stephenb ,

On paid support, Elastic support team told us that please raised this concern on community forum, that's why we have raised this query here.

Can you please tell us the complete steps about how to use codec.

We will again look into the shared document and will let you know.

Regards,
Syed

13

Ahhh That is probably because your subscription level does not offer guided support. Only break fix... Nonetheless, this is still a community forum with no SLAs or promises.

I can refer you to the docs But you will need to look at them and figure out.

But the bottom line is if you have that high level rate of authentications and you want to audit each and every authentication, then it's going to take whatever space it's going to take so that you can have your audit logs. There's no magic fix.

Set best_compression in the auditbeat template.

And drop processor the host / agent fields

processors:
  - drop_fields:
      fields: ["agent", "host", ...]
1 Like
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.