# 6.4 xpack.security.audit.logfile.events.ignore\_filters.\*.realms broken?

**URL:** <https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758>\
**Category:** Elasticsearch\
**Created:** [December 6, 2018, 4:02pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758 "2018-12-06T16:02:02Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmisztal1980](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmisztal1980/32/38519_2.png) [@mmisztal1980](https://discuss.elastic.co/u/mmisztal1980)\
**Post date:** [December 6, 2018, 4:02pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/1 "2018-12-06T16:02:03Z")

</div>

In our cluster, we have added the following snippet to our `elasticsearch.yaml` across all nodes:

```auto
xpack.security.audit.enabled: true
xpack.security.audit.outputs: [index]
xpack.security.audit.index.events.emit_request_body: true

```

We have restarted all nodes in the cluster and applied the following via the Kibana's dev-tools:

```auto
PUT /_cluster/settings 
{
    "transient": {
       
        "xpack.security.audit.logfile.events.ignore_filters": {
            "ourpolicyname": {
                "realms": [
                    "__attach",
                    "__anonymous",
                    "reserved"
                ]
            }
        }
    }
}

```

We have removed the daily `.security_audit_log-*` index and waited for it to become recreated.

Upon recreation our expectation was to see the ` __attach` , `__ anonymous` and `reserved` realms filtered out of the audit log. Instead, what we're seeing is:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/b/6bdc9eadf28e0d4ade0cb6d5063675e24c5b3e87.png)

Which indicates that events from the forementioned realms keep flowing in and that the audit log is not filtering them out as we intended. Our intention is to have the audit log ignore all of the events from those realms from all indices.

We have also tried to apply the equivalent `ignore_filters` configuration via `elasticsearch.yml` with the same result.

Please tell us if this is a miss-configuration on our end or is there a bug in this piece of functionality.

Best regards

---

<div class="post-metadata">

**Author:** ![Albert\_Zaharovits](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/albert_zaharovits/32/24390_2.png) [@Albert\_Zaharovits](https://discuss.elastic.co/u/Albert_Zaharovits)\
**Post date:** [December 10, 2018, 4:24pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/2 "2018-12-10T16:24:41Z")

</div>

Hi @mmisztal1980,

As the setting name `xpack.security.audit.logfile.events.ignore_filters` suggests the [ignore\_filters](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html#audit-event-ignore-policies) capability is available only for the [logfile audit output type](https://www.elastic.co/guide/en/elastic-stack-overview/6.5/audit-log-output.html) .

We are working on moving away from the index audit output as there are design limitations we cannot overcome otherwise. For example, it is too complicated to throttle calls generating audit events when the indexing throughput fluctuates. We recommend using the logfile auditing and we will have a convenient inside the stack solution for indexing it, soon...

---

<div class="post-metadata">

**Author:** ![mmisztal1980](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmisztal1980/32/38519_2.png) [@mmisztal1980](https://discuss.elastic.co/u/mmisztal1980)\
**Post date:** [December 10, 2018, 8:05pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/3 "2018-12-10T20:05:08Z")

</div>

Hi @Albert_Zaharovits, thanks for the reply.

Our requirement is to persist that within an index, so we will try to use filbeat to ingest the auditlog after filtering out the unwanted realms. That ought to do it, however we are still missing an ingestion pipeline.

Can you point me to one?  
Best regards

---

<div class="post-metadata">

**Author:** ![Albert\_Zaharovits](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/albert_zaharovits/32/24390_2.png) [@Albert\_Zaharovits](https://discuss.elastic.co/u/Albert_Zaharovits)\
**Post date:** [December 10, 2018, 9:43pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/4 "2018-12-10T21:43:47Z")

</div>

Using filebeat is the right strategy. The out-of-the-box-solution for this problem would be something very similar to what you're describing.

I recommend you upgrade to ES 6.5 as the [audit logfile is structured](https://www.elastic.co/guide/en/elastic-stack-overview/6.5/audit-log-output.html#audit-log-entry-format) . The one line JSON per line format should be easy for Filebeat to digest and then ship to ES (using a pipeline or not). You'd still have to define the [index template yourself](https://www.elastic.co/guide/en/elastic-stack-overview/6.5/audit-log-output.html#audit-log-entry-format) , and for inspiration you can look into the `log4j2.properties` file for the audit logfile appender (in ES).

We are working to making this process an out of the box experience, but we're not there yet...

---

<div class="post-metadata">

**Author:** ![mmisztal1980](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmisztal1980/32/38519_2.png) [@mmisztal1980](https://discuss.elastic.co/u/mmisztal1980)\
**Post date:** [December 11, 2018, 10:21am UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/5 "2018-12-11T10:21:27Z")

</div>

@Albert_Zaharovits we're currently running ES 6.4, does that support the same feature as 6.5?

---

<div class="post-metadata">

**Author:** ![Albert\_Zaharovits](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/albert_zaharovits/32/24390_2.png) [@Albert\_Zaharovits](https://discuss.elastic.co/u/Albert_Zaharovits)\
**Post date:** [December 11, 2018, 2:43pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/6 "2018-12-11T14:43:09Z")

</div>

Structured audit log was introduced in 6.5 .

---

<div class="post-metadata">

**Author:** ![mmisztal1980](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmisztal1980/32/38519_2.png) [@mmisztal1980](https://discuss.elastic.co/u/mmisztal1980)\
**Post date:** [December 12, 2018, 12:01pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/7 "2018-12-12T12:01:22Z")

</div>

Got it, thank you. We are preparing to upgrade one of our clusters to 6.5 to try this feature out.

---

<div class="post-metadata">

**Author:** ![mmisztal1980](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmisztal1980/32/38519_2.png) [@mmisztal1980](https://discuss.elastic.co/u/mmisztal1980)\
**Post date:** [December 18, 2018, 4:22pm UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/8 "2018-12-18T16:22:16Z")

</div>

Hi @Albert_Zaharovits, we've upgraded to 6.5.3, enabled to logfile output and are ingesting it with filebeat.

This is what filebeat produces so we'll need to use an ingestion pipeline with a json processor to extract the message and add it to the root of the document.

```json
{
  "_index": "auditlog-2018.12.18",
  "_type": "doc",
  "_id": "IxIRwmcB_DAbTiSg2aMw",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2018-12-18T16:06:47.880Z",
    "host": {
      "name": "Elastic01"
    },
    "source": "D:\\logs\\elasticsearch\\MachX-Test_audit.log",
    "offset": 104236160,
    "message": "{\"@timestamp\":\"2018-12-18T17:06:46,574\", \"node.name\":\"elastic01.schultzdev.local\", \"node.id\":\"VlN-m2IVR2-GsIkcGtY_pQ\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"user.name\":\"mam@schultz.dk\", \"user.realm\":\"schultzms\", \"user.roles\":[\"superuser\"], \"origin.type\":\"rest\", \"origin.address\":\"10.32.219.52:55485\", \"action\":\"cluster:admin/xpack/security/user/authenticate\", \"request.name\":\"AuthenticateRequest\"}",
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "Elastic01",
      "version": "6.4.0",
      "name": "Elastic01"
    }
  },
  "fields": {
    "@timestamp": [
      "2018-12-18T16:06:47.880Z"
    ]
  },
  "sort": [
    1545149207880
  ]
}

```

While doing early work with the ingestion pipeline:

```auto
PUT _ingest/pipeline/audit-log-ingestion-pipeline
{
  "description": "Audit Log Ingestion Pipeline Q4 2018",
  "processors": [
    {
      "json" : {
        "field" : "message",
        "target_field" : "msg"
      }
    }
  ],
  "on_failure": [
      {
        "set": {
          "field": "_index",
          "value": "audit-log-ingestion-pipeline-errors"
        }
      },
      {
        "set": {
          "field": "error",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ]
}

```

I've attempted to use the `simulate` api:

```auto
POST _ingest/pipeline/audit-log-ingestion-pipeline/_simulate
{
  "docs": [
    {
      "_index": "auditlog-2018.12.18",
      "_type": "doc",
      "_id": "IxIRwmcB_DAbTiSg2aMw",
      "_version": 1,
      "_score": null,
      "_source": {
        "@timestamp": "2018-12-18T16:06:47.880Z",
        "host": {
        "name": "Elastic01"
      },
      "source": "D:\\logs\\elasticsearch\\MachX-Test_audit.log",
      "offset": 104236160,
      "message": "{\"@timestamp\":\"2018-12-18T17:06:46,574\", \"node.name\":\"elastic01.schultzdev.local\", \"node.id\":\"VlN-m2IVR2-GsIkcGtY_pQ\", \"event.type\":\"transport\", \"event.action\":\"access_granted\", \"user.name\":\"mam@schultz.dk\", \"user.realm\":\"schultzms\", \"user.roles\":[\"superuser\"], \"origin.type\":\"rest\", \"origin.address\":\"10.32.219.52:55485\", \"action\":\"cluster:admin/xpack/security/user/authenticate\", \"request.name\":\"AuthenticateRequest\"}",

      "input": {
        "type": "log"
      },
      "prospector": {
        "type": "log"
      },
      "beat": {
        "hostname": "Elastic01",
        "version": "6.4.0",
        "name": "Elastic01"
      }
    },
    "fields": {
      "@timestamp": [
        "2018-12-18T16:06:47.880Z"
      ]
    },
    "sort": [
      1545149207880
    ]
  }]
}

```

And go an interesting error message in return:

```json
{
  "error": {
    "root_cause": [
      {
        "type": "class_cast_exception",
        "reason": "java.lang.Integer cannot be cast to java.lang.Long"
      }
    ],
    "type": "class_cast_exception",
    "reason": "java.lang.Integer cannot be cast to java.lang.Long"
  },
  "status": 500
}

```

Tried it with message equal to the equivalent of :

```json
{ "x": "y" }

```

And got the same error.

Can you advise what is wrong here?

---

<div class="post-metadata">

**Author:** ![Albert\_Zaharovits](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/albert_zaharovits/32/24390_2.png) [@Albert\_Zaharovits](https://discuss.elastic.co/u/Albert_Zaharovits)\
**Post date:** [December 20, 2018, 11:10am UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/9 "2018-12-20T11:10:49Z")

</div>

Hi @mmisztal1980

The filebeat output document doesn't look right, but I'm no expert to say what's missing (more so because of not posting the configuration). For an example see the [structured-logging-filebeat](https://www.elastic.co/blog/structured-logging-filebeat) blog post, even though it's a bit dated.  
For setting reference of the filebeat log input see [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#filebeat-input-log-config-json)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2019, 11:11am UTC](https://discuss.elastic.co/t/6-4-xpack-security-audit-logfile-events-ignore-filters-realms-broken/159758/10 "2019-01-17T11:11:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.