# Kibana automatic activity is flooding audit log **URL:** \ **Category:** Kibana\ **Created:** [March 21, 2017, 12:40pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413 "2017-03-21T12:40:04Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Guy\_Shilo](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@Guy\_Shilo](https://discuss.elastic.co/u/Guy_Shilo)\ **Post date:** [March 21, 2017, 12:40pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/1 "2017-03-21T12:40:04Z")
Hello I enabled auditing in my cluster using x-pack. In the log file I can see that Kibana is constantly sending monitoring and health check requests to the cluster, even when there is no user activity. Every few seconds I get a bunch of messages like this: [2017-03-20T23:08:04,587] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/main], request=[MainRequest] [2017-03-20T23:08:04,592] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/nodes/info], request=[NodesInfoRequest] [2017-03-20T23:08:04,593] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/nodes/info[n]], request=[NodeInfoRequest] [2017-03-20T23:08:04,609] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/nodes/info], request=[NodesInfoRequest] [2017-03-20T23:08:04,610] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/nodes/info[n]], request=[NodeInfoRequest] [2017-03-20T23:08:04,614] [transport] [access\_granted] origin\_type=[rest], origin\_address=[192.168.1.240], principal=[kibana], action=[cluster:monitor/health], indices=[.kibana], request=[ClusterHealthRequest] This clutters the log and making it hard to find the really important messages. On the other hand, I cannot just filter those messages out based on IP address or user name, since I want Kibana "real" activity created by users to be caught by the audit process. Does anyone know how to stop those messages from appearing in the audit log file ? Thanks Guy ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [March 22, 2017, 6:14pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/2 "2017-03-22T18:14:59Z")
If you are on 5.0+ there is this option: Modify the config/x-pack/log4j2.properties file. Add the following to the file and restart the ES node(s). appender.audit\_rolling.filter.regex.type = RegexFilter appender.audit\_rolling.filter.regex.onMatch = DENY appender.audit\_rolling.filter.regex.regex = ._principal=\[Kibana\]._|._indices=\[.monitoring-data-2\]._ appender.audit\_rolling.filter.regex.onMisMatch = ACCEPT The above example denies any log entries for the Kibana user as well as access to the .monitoring-data-2 index. Note: Syntax above is case-sensitive, and double escaping is required for the [and] (We have an outstanding enhancement request to provide more native filtering capabilities for X-pack Security) This an example from one of our engineers, but I think you can modify it to your needs from here. 🙂 ---
**Author:** ![Guy\_Shilo](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@Guy\_Shilo](https://discuss.elastic.co/u/Guy_Shilo)\ **Post date:** [March 23, 2017, 10:01am UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/3 "2017-03-23T10:01:41Z")
Hello I will give it a try. Too bad x-pack does not have it built in, It should consider Kibana monitoring and not treat it as standard event. ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [March 23, 2017, 11:18am UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/4 "2017-03-23T11:18:26Z")
There is an enhancement request for this, I found it yesterday, but I don't know when it will be picked up. ---
**Author:** ![jonathansylvain](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jonathansylvain](https://discuss.elastic.co/u/jonathansylvain)\ **Post date:** [March 23, 2017, 2:11pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/5 "2017-03-23T14:11:52Z")
Hi Marius, That config worked like a charm for the log files, that'll save some disk space. Is there an equivalent for the audit logs indexed in ES rather than a logfile? Thanks JS ---
**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\ **Post date:** [March 23, 2017, 2:55pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/6 "2017-03-23T14:55:55Z")
Oh, there is. You need to add this to the elasticsearch.yml file. `xpack.security.audit.outputs: [index, logfile]` But the filter will only work for the log file, although you can browse them a lot more easily in ES + Kibana. There are a few drawbacks to logging to a index only, this is why with the options from before it will log both to file and ES. Read more about them here (it's in the first article of the page): [https://www.elastic.co/guide/en/x-pack/current/auditing.html](https://www.elastic.co/guide/en/x-pack/current/auditing.html) ---
**Author:** ![jonathansylvain](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@jonathansylvain](https://discuss.elastic.co/u/jonathansylvain)\ **Post date:** [March 23, 2017, 3:23pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/7 "2017-03-23T15:23:35Z")
Hi Marius, Thank you for the prompt response. I already have audit log sent to index using the method described above, and i've added exclusions for authentication\_success, however i cannot do so for the access\_granted since the messages i'm most interested in other than failed auith comes from those messages (deletion events) This in turn generates 3.2 to 3.3 million events per day for the kibana and logstash users alone, since i need a minimum retention of 90 days for active logs and a year for archived logs that's a lot of clutter. If there are no filtering options for the audit logs sent directly index other than events exclusion would it be something that's included in the enhancement request you've described above or would a new request need to be created. In the meanwhile i'm thinking i'll either create a daily cornt job to perform a delete\_by\_query or simply ingest the logfile through logstash. Thanks JS ---
**Author:** ![Guy\_Shilo](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@Guy\_Shilo](https://discuss.elastic.co/u/Guy_Shilo)\ **Post date:** [March 25, 2017, 8:19pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/8 "2017-03-25T20:19:33Z")
Hello It doesn't seem to work for me. I tried the very same regular expression you suggested, and it did not like the hyphens (even when I escaped them) and threw errors. I tried a little different regular expression: .principal=[Kibana].|.action=[.monitoring.\*] And it just ignored it and kept on flooding the log with audit messages that were supposed to be filtered out. What am I doing wrong ? ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [March 26, 2017, 10:59pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/9 "2017-03-26T22:59:54Z")
> `principal=[Kibana].|.action=[.monitoring.*]` That needs to be a Java [regular expression](https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html), and the `[` character has special meaning there. The trouble with escaping is that you have deal with properties files and regular expressions, so some things need multiple escapes. The simplest expression that doesn't need any escaping would be: ``` principal=.Kibana.|.indices=..monitoring-data-2.. ``` There's some improvements that could be made, but that should do the job for you. ---
**Author:** ![Guy\_Shilo](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@Guy\_Shilo](https://discuss.elastic.co/u/Guy_Shilo)\ **Post date:** [March 27, 2017, 10:35am UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/10 "2017-03-27T10:35:19Z")
Hello I am not that great with regular expressions, however, trying to filter out Kibana monitoring messages, I came up with this regexp: principal=.Kibana.|.indices=..(monitoring-data-2|kibana)..|action=.\*cluster:monitor\* I tested it with an [online Java regexp tester](http://www.freeformatter.com/java-regex-tester.html) and it was good and matched all the required messages. But when I inserted it to log4j2.properties and restarted, it seems like Elasticsearch is completely ignoring it ! I keep seeing the messages in the audit file... Any ideas ? Thanks Guy ---
**Author:** ![jetnet](https://avatars.discourse-cdn.com/v4/letter/j/a87d85/32.png) [@jetnet](https://discuss.elastic.co/u/jetnet)\ **Post date:** [March 27, 2017, 11:21am UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/11 "2017-03-27T11:21:42Z")
it looks like the regex has to macht the WHOLE string, not just a part of it, the following works for me: ``` appender.audit_rolling.filter.regex.regex = .*principal=.elastic...action=.indices:data/write/bulk.*|.*principal=.kibana.,.* ``` it filters out all `kibana` user requests and the following index requests from `elastic` user: ``` [2017-03-27T13:01:30,486] [transport] [access_granted] origin_type=[rest], origin_address=[::1], principal=[elastic], action=[indices:data/write/bulk[s][p]], request=[ConcreteShardRequest] ``` Please try it out! ---
**Author:** ![Guy\_Shilo](https://avatars.discourse-cdn.com/v4/letter/g/c0e974/32.png) [@Guy\_Shilo](https://discuss.elastic.co/u/Guy_Shilo)\ **Post date:** [March 27, 2017, 1:28pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/12 "2017-03-27T13:28:53Z")
Hello This is what eventually eliminated all unwanted messages: .\*principal=.kibana...action=.cluster:monitor.\*|.\*action=.cluster:admin.\*|.\*indices=..kibana.,.\*|.\*indices=..\*.,.\* Thank you all for your help Guy ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 24, 2017, 1:29pm UTC](https://discuss.elastic.co/t/kibana-automatic-activity-is-flooding-audit-log/79413/13 "2017-04-24T13:29:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.