How to apply different filter for different column in a single Kibana visualisation (Data Table)?

Elastic Stack Kibana
1

Hi All,

I have created 2 index say a-b-middletier and a-b-database upon that i have created a kibana visualisation (Data Table) by merging these 2 indexes say a-b-*. i would like to see the counts w.r.t middletier and database with the label say count_for_middletier and count_for_database.

Expected result:
Name count_for_database count_for_middletier
middletier -/0 100
database 100 -/0

But am getting the following:

Name count_for_database count_for_middletier
middletier 100 100
database 100 100

It would be very helpful, if you can help me on how to put different filters on counts(aggregation fields) ??

And also can we have different filter on different aggregate columns example count, sum, average..??

Thanks
Pramod

1 Like
2

Hi Pramod,

There might be other ways to do this, but I found a way to get those counts only for specific indexes (not wildcarded);
I have packetbeat and topbeat running. I created an index pattern *beat-* which is what I've used in this Data Table visualization. I Split Rows on a Filters aggregation type, and added 2 filters;
_index: topbeat-2016.08.24
and
_index: packetbeat-2016.08.24

2016-08-24 12_53_36-ubuntu64 (build OS packages works) [Running] - Oracle VM VirtualBox.png1792×1067 167 KB

Regards,
Lee

Also see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-index-field.html
which describes that _index is a virtual field. It's not something that is an actual indexed field in the docs.

3

I found another way that doesn't require using the exact index names but it's a bit more hacky.

  1. Create a visualization Split Rows on Terms, and pick any string type of field.
  2. Save that visualization
  3. go to Settings > Objects > Visualization tab and click on the visualization you just saved
  4. go down to the visState and replace that string field from step 1 with _index
  5. Save your changes
  6. You can get directly to your visualization by clicking the eye icon next to your visualization name

2016-08-24 13_44_10-ObjectEditor.png682×580 91.7 KB

One issue with this is that when you look at the Data tab for the visualization it shows a blank for the Field because _index isn't in the list so it can't show that selection. This technique will find all indexes that match your current index pattern, but it's not going to group them by their type like a-b-middletier or a-b-database (I'm assuming you have time-based indexes).

2016-08-24 13_42_20-DataTableNotShowing_index.png1808×1219 161 KB

I think is the same issue as with _type and I'll comment on this issue;

Regards,
Lee

4

Thanks a lot for the help LeeDr.

I would like to tell you the whole scenario, so that you can help me resolving this issue in particular.
I have 2 index, one for database and another for weblogic.
I will be getting 2 different set of fields i.e one set for middletier and the other for database right and there are few common fields like name and version among both.

Example: Lets assume you got 2 products product 1 and 2 with some version, both product have some middletier and database events you want to list(count). can you Please help me how to resolve this issue ??

I would like to see 4 following columns (expected).
Name Version count(middletier) count(database)
product_1 1.1 25 20
product_2 1.2 34 12

It will be really very helpful, if you can help me resolving this issue.

Thanks
Pramod

5

yes, this is time based indexing.

Thanks
Pramod

6

Hi Pramod,

I don't understand the rest of your question regarding "different filters on counts(aggregation fields)". Could you explain more?

Thanks,
Lee

7

Sure, please let me know if you any doubt, i will narrate with some other example, but please help me with this issue.

assume we are working on 2 different product with different versions.

Example: Lets assume we have 4 indexes
a-b-database-product_x [Fields => name, version,....]
a-b-database-product_y [Fields => name, version,....]
a-b-middletier-product_x [Fields => name, version,....]
a-b-middletier-product_y [Fields => name, version,....]

with some common fields (name and version) among all 4 indexes.
so here product_1 will have both middletier and database events (index: a-b-database-product_x, a-b-middletier-product_x)
hence the product_1 has count of 25 for middletier and 20 for the the database right, so for product_y ?

If i use the index as a-b-* in the kibana visualisation (very specifically Data Table), i would expect the following to appear.

I would like to see 4 following columns (expected).

Name                 Version       count(middletier)    count(database)

product_x 1.1 25 20
product_y 1.2 34 12

It will be really very helpful, if you can help me resolving this issue.??
please let me know if you have any doubt.

Thanks
Pramod

8

I hope you got my question by now, otherwise please let me know.(pramodkumar.cu@gmail.com)

Thanks in advance
Pramod

9

Hi Pramod,

If you followed along with my previous steps and have a Data Table which has a terms aggregation on _index, then you should click to "Add sub-buckets" under that and again select Terms, version.

It won't lay it out exactly as you have shown like a spreadsheet. Instead it will have one column for all counts, and the rows will be for each unique set of the aggregations you have.

Thanks,
Lee

10

Can't we have multiple count columns in one Data Table (kibana Visualisation) and specific filter for each column say count(middletier) or count(database)??

I was thinking, can we do this with JSON input under advanced option ??

Thanks
Pramod

1 Like
11

The current Data Table visualization doesn't support that. There's at least one enhancement request asking for it;

12

If JSON input is the solution, could you tell me how to put filter for some regex inside the JSON input ?

kibana1.png392×600 12.8 KB
kibana.png1136×272 3.86 KB

Thanks
Pramod

13

Any Idea, when will that enhancement will be delivered to customers ??

Thanks
Pramod

14

No estimate. It doesn't seem that hard, but I'm not a developer familiar with that code.