# Elasticsearch audit log take more size

**URL:** <https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [November 23, 2022, 8:17am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627 "2022-11-23T08:17:37Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\
**Post date:** [November 23, 2022, 8:17am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/1 "2022-11-23T08:17:37Z")

</div>

Hi Team,

Elasticsearch audit log take 20-25 GB size every day. I'm using elasticsearch 7.16.2 and same version for filebeat. I have enable the audit key by enabling the below keys in elasticsearch.yml file.

```auto
xpack.security.enabled: true
xpack.security.audit.enabled: true
xpack.security.audit.logfile.events.emit_request_body: true
xpack.security.audit.logfile.emit_node_name: true
xpack.security.audit.logfile.events.include: ["authentication_success"]

```

Below I have added filebeat.yml file configuration.

```auto
###################### Filebeat Configuration #########################

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# filestream is an input for collecting log messages from files.
- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   - D:\Elastic\logs\elasticsearch_audit-*.json

```

Can you help me to minimize the size.

---

<div class="post-metadata">

**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\
**Post date:** [November 23, 2022, 6:37pm UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/2 "2022-11-23T18:37:30Z")

</div>

Hi Team,

Elastic search audit log take more space. What I can do for this so that audit log file size will reduce? Please help me.

Thanks,  
Nitin Bisht

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 23, 2022, 6:45pm UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/3 "2022-11-23T18:45:45Z")

</div>

> [@Nitin08bisht](#):
>
> `xpack.security.audit.logfile.events.emit_request_body: true`

This setting is very verbose... Which is probably adding significantly to the size of your logs

Do you really need that?

---

<div class="post-metadata">

**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\
**Post date:** [November 24, 2022, 4:40am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/4 "2022-11-24T04:40:53Z")

</div>

Hi @stephenb ,

I'm going to set the mention key as false. And I will let you know what is the result after getting one day log.

```auto
xpack.security.audit.logfile.events.emit_request_body: false

```

Thanks,  
Nitin Bisht

---

<div class="post-metadata">

**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\
**Post date:** [November 24, 2022, 9:46pm UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/5 "2022-11-24T21:46:48Z")

</div>

> [@Nitin08bisht](#):
>
> `xpack.security.audit.logfile.events.emit_request_body: false`

@Nitin08bisht This will definitely have a big effect. This setting will log the full request body of every write and read, which results in more then double the amount of data then your ingesting. When I enable this, my Elastic cluster explodes lol.

Unfortunately this setting enables extensive auditing, which sometimes is needed for compliance etc. If only Elastic would give us the ability to enable this auditing setting only for some high sensitive indices.

Willem

---

<div class="post-metadata">

**Author:** ![Nitin08bisht](https://avatars.discourse-cdn.com/v4/letter/n/7cd45c/32.png) [@Nitin08bisht](https://discuss.elastic.co/u/Nitin08bisht)\
**Post date:** [November 25, 2022, 6:55am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/6 "2022-11-25T06:55:25Z")

</div>

Hi @stephenb /@willemdh

Thanks for the solution. It really help me a lot. The size of elasticsearch auditlogs has reduced considerably as compared to earlier.

Before applying the below keys in elasticsearch.yml file the elasticsearch auditlogs file take too much size. But when I applied the below key, then it reduced the size of elasticsearch auditlogs file.

```auto
xpack.security.audit.logfile.events.emit_request_body: false

```

Thanks,  
Nitin Bisht

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 26, 2022, 1:01am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/7 "2022-11-26T01:01:47Z")

</div>

Hi @willemdh

Yes agreed Elastic needs to provide better/ easier audit logging.

> [@willemdh](#):
>
> This setting will log the full request body of every write and read, which results in more then double the amount of data then your ingesting.

Yup totally true if you gave \_all event types.

One thing you can do, is in the filebeat audit module add drop events that you are not interested does not contain the indices you are interested... not great but can be done.. there or in a/the ingest pipeline

---

<div class="post-metadata">

**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\
**Post date:** [November 26, 2022, 10:18am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/8 "2022-11-26T10:18:23Z")

</div>

@stephenb Thanks for the suggestion, but imho that's not an ideal solution for multiple reasons, mostly related to unnecessary load (big envs). Atm I'm filtering my audit logs in a Logstash pipeline, because the available filtering options in Filebeat are not granular enough tbh.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 26, 2022, 3:12pm UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/9 "2022-11-26T15:12:35Z")

</div>

Totally agree... Not ideal... Logstash was not mentioned, that is good place to do the filtering... Better will be when elasticsearch provides better native granularity.

---

<div class="post-metadata">

**Author:** ![syed0510](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@syed0510](https://discuss.elastic.co/u/syed0510)\
**Post date:** [December 12, 2022, 7:16am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/10 "2022-12-12T07:16:41Z")

</div>

Hi @stephenb /@willemdh ,

Audit logs are still taking 20gb size per day after set mentioned key as **false**. Requesting you to please look into this issue as me and Nitin are in the same team and facing the same issue.

Regards,  
Syed

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2022, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/11 "2022-12-12T07:32:17Z")

</div>

Have you looked at the actual logs and confirmed only the events you want are being logged.

Second @syed0510 this is a community forum not paid support.

> [@Nitin08bisht](#):
>
> Thanks for the solution. It really help me a lot. The size of elasticsearch auditlogs has reduced considerably as compared to earlier

I am confused this seems to indicate the logs reduced considerably.

I suggest you look at the audit settings.

> **[Auditing security settings | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html)**

Other things you could does is set the codec to best compress and / or drop some of the repeated host / agent fields with a processor in auditbeat.

There is no magic you have control... In general Elasticsearch auditing is pretty verbose...

---

<div class="post-metadata">

**Author:** ![syed0510](https://avatars.discourse-cdn.com/v4/letter/s/76d3ee/32.png) [@syed0510](https://discuss.elastic.co/u/syed0510)\
**Post date:** [December 12, 2022, 7:47am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/12 "2022-12-12T07:47:59Z")

</div>

Hi @stephenb ,

> [@stephenb](#):
>
> this is a community forum not paid support.

On paid support, Elastic support team told us that please raised this concern on community forum, that's why we have raised this query here.

> [@stephenb](#):
>
> Other things you could does is set the codec to best compress and / or drop some of the repeated host / agent fields with a processor in auditbeat.

Can you please tell us the complete steps about how to use codec.

We will again look into the shared document and will let you know.

Regards,  
Syed

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 12, 2022, 8:07am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/13 "2022-12-12T08:07:01Z")

</div>

> [@syed0510](#):
>
> On paid support, Elastic support team told us that please raised this concern on community forum, that's why we have raised this query here.

Ahhh That is probably because your subscription level does not offer guided support. Only break fix... Nonetheless, this is still a community forum with no SLAs or promises.

> [@syed0510](#):
>
> Can you please tell us the complete steps about how to use codec.

I can refer you to the docs But you will need to look at them and figure out.

But the bottom line is if you have that high level rate of authentications and you want to audit each and every authentication, then it's going to take whatever space it's going to take so that you can have your audit logs. There's no magic fix.

Set [best\_compression](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-codec) in the auditbeat template.

And [drop processor](https://www.elastic.co/guide/en/beats/auditbeat/current/drop-fields.html) the host / agent fields

```auto
processors:
  - drop_fields:
      fields: ["agent", "host", ...]
   

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 9, 2023, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-audit-log-take-more-size/319627/14 "2023-01-09T10:07:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.