# .security\* log has millions of record with 'kibana' principle accessing empty .reporting-$date$ indices

**URL:** <https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166>\
**Category:** Kibana\
**Tags:** elastic-stack-security\
**Created:** [April 10, 2019, 8:18am UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166 "2019-04-10T08:18:06Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mje/32/42234_2.png) [@mje](https://discuss.elastic.co/u/mje)\
**Post date:** [April 10, 2019, 8:18am UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/1 "2019-04-10T08:18:06Z")

</div>

Our 'kibana' principle figures in 20-80+ million log records pr day in our .security\* xpack/security indices daily, related to searches in .reporting-$date indices, constantly.

We dont have any report jobs, and the contents of the existing .reporting\* indices is \< 50 docs per index. We had 8 .reporting indices, of various dates.

How do i find out why Kibana is searching there so frequently that it's trashing the .security\* indices? It generates quite the log volume. Also, naturally, the text log is getting quite big.

I have a snapshot of the indices; however I've deleted the .reporting indices, and that has dropped the rate of logging into the .audit\* indices dramatically.

However, when I look into the Monitoring dashboards in Kibana, the drop in indexing rate into the .security$date index isn't matching the drop in reported searches (they might be cumulatively/bulk indexed).

Why is this excessive search (and consequentual logging of searches) into .reporting indices happening?

Also, I see logs related to .monitoring indices in excessive amounts (millions), when looking at the Monitoring dashboard, even for a short duration of time.

Can some of this logging be filtered out so it does'nt hit the .security index and log file? If not, can I disable Security logging of access to the .reporting or .monitoring indices?

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [April 12, 2019, 10:23am UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/2 "2019-04-12T10:23:28Z")

</div>

Those requests are kibana checking for any reporting jobs that are or are not active. As for the fact that it's saved as an event in the .security index, do you have audit enabled?  
I know there is a way to filter out some events from the ES audit, but not sure about Kibana audit logs. I can look into it if you're using Kibana audit.

---

<div class="post-metadata">

**Author:** ![mje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mje/32/42234_2.png) [@mje](https://discuss.elastic.co/u/mje)\
**Post date:** [April 12, 2019, 1:40pm UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/3 "2019-04-12T13:40:55Z")

</div>

Hi Marius,

We are migrated from X-Pack security in 5.x to 6.7, and we are using audit as well as security yes.

I'd really like to hint which indices I'd like auditing on, that would help me tremendously.

What puzzles me is, that Kibana generates between 25 and 80 million records in the .security\* log every 24 hours - and that is only those entries regarding access to 5-8 .reporting indices, all practically empty (8 documents most...).

I appreciate the fact that Kibana looks for old reporting activity and settings, but thousand of times a second 24/7? That sounds more like a bug. 🙂

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [April 12, 2019, 1:56pm UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/4 "2019-04-12T13:56:13Z")

</div>

There have been changes with that recently, It shouldn't be that large.  
As for solving your problem, i'd recommend filtering out these events from the audit logs with a setting like this:  
`xpack.security.audit.logfile.events.ignore_filters.<policy_name>.users`

[https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html)

---

<div class="post-metadata">

**Author:** ![mje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mje/32/42234_2.png) [@mje](https://discuss.elastic.co/u/mje)\
**Post date:** [April 12, 2019, 2:03pm UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/5 "2019-04-12T14:03:18Z")

</div>

Thank you for your help.

It appears that what you link to is regarding the new .json log files; I'm seing this in the .security\_audit\_log-$date indices - or am I wrong?

If so; will the filtering policy for the .json log files translate to same behavior in the index-output?

---

<div class="post-metadata">

**Author:** ![Marius\_Dragomir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_dragomir/32/42087_2.png) [@Marius\_Dragomir](https://discuss.elastic.co/u/Marius_Dragomir)\
**Post date:** [April 12, 2019, 2:08pm UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/6 "2019-04-12T14:08:45Z")

</div>

There are multiple output types available for security audit: the json and ES indexing. I think you fall in the second category.  
[https://www.elastic.co/guide/en/elasticsearch/reference/6.7/auditing-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.7/auditing-settings.html)  
That link was for 7.0, this one is the correct one, sorry.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2019, 2:08pm UTC](https://discuss.elastic.co/t/security-log-has-millions-of-record-with-kibana-principle-accessing-empty-reporting-date-indices/176166/7 "2019-05-10T14:08:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.