.security* log has millions of record with 'kibana' principle accessing empty .reporting-$date$ indices

Elastic Stack Kibana
1

Our 'kibana' principle figures in 20-80+ million log records pr day in our .security* xpack/security indices daily, related to searches in .reporting-$date indices, constantly.

We dont have any report jobs, and the contents of the existing .reporting* indices is < 50 docs per index. We had 8 .reporting indices, of various dates.

How do i find out why Kibana is searching there so frequently that it's trashing the .security* indices? It generates quite the log volume. Also, naturally, the text log is getting quite big.

I have a snapshot of the indices; however I've deleted the .reporting indices, and that has dropped the rate of logging into the .audit* indices dramatically.

However, when I look into the Monitoring dashboards in Kibana, the drop in indexing rate into the .security$date index isn't matching the drop in reported searches (they might be cumulatively/bulk indexed).

Why is this excessive search (and consequentual logging of searches) into .reporting indices happening?

Also, I see logs related to .monitoring indices in excessive amounts (millions), when looking at the Monitoring dashboard, even for a short duration of time.

Can some of this logging be filtered out so it does'nt hit the .security index and log file? If not, can I disable Security logging of access to the .reporting or .monitoring indices?

2

Those requests are kibana checking for any reporting jobs that are or are not active. As for the fact that it's saved as an event in the .security index, do you have audit enabled?
I know there is a way to filter out some events from the ES audit, but not sure about Kibana audit logs. I can look into it if you're using Kibana audit.

3

Hi Marius,

We are migrated from X-Pack security in 5.x to 6.7, and we are using audit as well as security yes.

I'd really like to hint which indices I'd like auditing on, that would help me tremendously.

What puzzles me is, that Kibana generates between 25 and 80 million records in the .security* log every 24 hours - and that is only those entries regarding access to 5-8 .reporting indices, all practically empty (8 documents most...).

I appreciate the fact that Kibana looks for old reporting activity and settings, but thousand of times a second 24/7? That sounds more like a bug. :slight_smile:

4

There have been changes with that recently, It shouldn't be that large.
As for solving your problem, i'd recommend filtering out these events from the audit logs with a setting like this:
xpack.security.audit.logfile.events.ignore_filters.<policy_name>.users

https://www.elastic.co/guide/en/elasticsearch/reference/current/auditing-settings.html

5

Thank you for your help.

It appears that what you link to is regarding the new .json log files; I'm seing this in the .security_audit_log-$date indices - or am I wrong?

If so; will the filtering policy for the .json log files translate to same behavior in the index-output?

6

There are multiple output types available for security audit: the json and ES indexing. I think you fall in the second category.
https://www.elastic.co/guide/en/elasticsearch/reference/6.7/auditing-settings.html
That link was for 7.0, this one is the correct one, sorry.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.