My challenge - Kibana visualization for complex reports

Elastic Stack Kibana
1

Hi there,

I'm pretty new with Kibana, so my "terms" could be wrong or inaccurate. I apologize in advance for that.

We are using a software that scans a share hosted on a NAS and logs 145 different metadata in an Elastic DB. From that list, I want to report specific metrics, with or without a query.

One report I have to build is: Files with rights that deny administrator access or a given user / group

So, I created a visualization (data table) with metrics "count" and buckets "split rows":

"title": "Share security list",
"type": "table",
"params": {
"perPage": 10,
"showPartialRows": false,
"showMetricsAtAllLevels": false,
"sort": {
"columnIndex": null,
"direction": null
},
"showTotal": true,
"totalFunc": "sum"
},
"aggs": [
{
"id": "1",
"enabled": true,
"type": "count",
"schema": "metric",
"params": {}
},
{
"id": "2",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "generalMetadata.localPath",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Local path"
}
},
{
"id": "3",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "cifsMetadata.fileSecurity.group.accountName",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Group"
}
},
{
"id": "4",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "cifsMetadata.fileSecurity.group.authorityName",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Domain"
}
},
{
"id": "5",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "cifsMetadata.fileSecurity.group.sidString",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Group SID"
}
},
{
"id": "6",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "cifsMetadata.owner",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Owner"
}
},
{
"id": "7",
"enabled": true,
"type": "terms",
"schema": "bucket",
"params": {
"field": "cifsMetadata.fileSecurity.owner.sidString",
"size": 5,
"order": "desc",
"orderBy": "_key",
"otherBucket": false,
"otherBucketLabel": "Other",
"missingBucket": false,
"missingBucketLabel": "Missing",
"customLabel": "Owner SID"
}
}
]
}

I'm pretty happy with this visualization, but I'm looking for 2 things:

  • Does Kibana have a size limit for a split row? (I could have to generate this visualization for more than a million of lines...)
  • How may I query this list to narrow it and report only "localpath" that doesn't contain a specific term?

For example, I have "Share A" with owners "Bob" and "Admin", and "share B" with the owner "Bob" only. It should have the owner "Admin" then, in theory, but it's not in real life... so, I'm looking to report/highlight this "share B". I hope you get what I mean!

Many thanks for your help

Cheers

G.

2

There is a size limit on term aggregation. (Size setting).

to narrow down query you probably want to apply filters, however you can only filter on per document basis (so you can't filter for all the records with owner bob, where there is no matching record with owner admin).

if you have one document per localPath and owners is an array, then you can try searching for ducuments that don't have a specific term (like "admin")
- add filter -> select field -> is not -> admin

3

Hey Peter,

Thanks for your reply, much appreciated!

Got it about how to filter, it was my understanding though... thanks for the confirmation.

Regarding the size limit on term aggregation, I can't find a clear answer. what's the max size we can set? May I set a size limit up to 1000000 if I want?

Thanks

4

probably not.

first constrain is elasticsearch, where term aggregation on such a large number of terms might (depending on your cluster and amount of data) cause problems.

the second constrain is your browser, which just won't be able to handle the response that big. Currently kibana does pagination in browser memory.

5

Excellent!

Thanks for the quick turnaround!

Have a terrific day

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.