Visualize data from multiple indexes in one chart

[Mehak_Bhargava]

Hi, I have 5-7 indexes with different names and want to create one chart which will get a field that is existing in all indexes. For example index 1, index 2, index3, have a field called "type" and I want to get the count of "type" in all indexes in Kibana's visualize app. I do not want to combine all indexes into one index, as I want to see the sepearte index names.

[LizaD (Liza Dayoub)]

Hi @Mehak_Bhargava,

Each visualization works on one index pattern, you can group index patterns by wild card like: index1, index2, index3, ...etc can have index pattern: index*. However seems like you want to see each index separately, so the other option is to create a visualization for each index pattern and add them to one dashboard. Hope this helps.

Thanks,
Liza

[Mehak_Bhargava]

Hi @LizaD, thanks for replying!
For index pattern, I am not sure if it will work because my index1 is called obapi, index2 is called rmmserver, and index3 is called dispatcher with all different names. Since they dont have similarity in naming, can one index pattern still be created for them? Because then I could make it work with my charts!

If not, is there a way that I can make a chart based on a field called type for instance, that occurs in all these three indexes? So aggregation will be solely based on field name!

I read this post which discussed that as long as I have field name same- "type" I can create a series?

How to plot multiple series by multiple indexes into one visualization?

in the panel settings of time series visual builder you can override the index pattern. you can even set it to * (to look at all index patterns). the only requirement is that all your index patterns have the same time field (for example @timestamp). then you can plot metrics on fields from different indexes. you could also do the same using timelion, without the restriction of same time field: .es(index=myfirstindex, timefield=timestamp, metric=sum:bytes), .es(index=secondindex, timefield=tim…

[LizaD (Liza Dayoub)]

Hi @Mehak_Bhargava,

Yes that is correct you can use time series visual builder on multiple index patterns, as yes in your case, the wildcard won't work since the names don't share a substring.
Hope that helps, give it a try and let us know. Thanks!

Liza

[Mehak_Bhargava]

The documentation for time series visualization for kibana 7.4 version isnt available. Is that still a feature?

Also, why do I have empty lines in kibana under under my index? Had to use a picture here to explain properly

Please explain or suggest where should I look ino to resolve this.

[LizaD (Liza Dayoub)]

Hi @Mehak_Bhargava,

Yes it should be available but it was renamed to TSVB in version 7.3.0 so please see guide here:
https://www.elastic.co/guide/en/kibana/7.4/TSVB.html

Do you mean the lines showing as '-' I believe that means no data is returned for that date/time for that field, can you confirm whether you do have data and it is not showing?

Thanks,
Liza

[Mehak_Bhargava]

Hi @LizaD, thank you for documentation. Let me get back to you on how it works out so others know it too.

And yes, I do have data. I have a file called dispatcher-adapter-quartz.logs and it has data and it appears in fields I have selected. But same again few lines above, it wont show data. if all data is read, does it send the "-" then? As you see below, timestamp and loglevel is extracted for the same file at below but not in above lines.

[LizaD (Liza Dayoub)]

Hi @Mehak_Bhargava,

Thanks please do let us know how TSVB works out or if you have any questions.

On the discover question, I checked some of my data and when I see the dash present it is on fields that are not applicable, when I traced back that data in Kibana to the source it matched in that those fields are not applicable.

Check your source data and mappings to see what the difference is between the events showing fields and the ones not showing fields. If there is data missing can you give me the sample source correlated.

Thanks,
Liza

[Mehak_Bhargava]

Hi @LizaD,

I got a chart where loglevel of all indexes are present. But As you see on the right hand side, the ledger has repetitive field like "INFO" and "INFO". One INFO has 3529 count but other INFO has 0. Why do we have repetitions?
I understand why "INFO" and "info'" will be repeated. but can I not mention that "INFO"= "info" so there is no repetition? And also remove the WARN =0 and INFO =0? And, I do have some loglevel under DEBUG, so I am not sure why that is showing 0 as well?

I traced the "-" and found that when the files on server havent updated, then these "-" comes in. For example, my dispacter.log was last updated on April 2nd, so after the logs were groked and fields displayed until April 2nd, the discover shows "-" from until last time file updated till now. I understand that this log aggregation is always looking for real time files but shouldnt there be a better way in discover to only show when files were updated and not show "-" when file isn't updated? Please suggest a permanent solution for it.

A temporary solution would be to remove the "@timestamp" from the step where we create the index pattern so the logs will be mapped to whenever data comes, right?

Thanks,
Mehak

[LizaD (Liza Dayoub)]

Thanks @Mehak_Bhargava, let me see if someone from our visualization team can help answer.

@flash1293 can you help?

[Mehak_Bhargava]

Hi @LizaD and @flash1293, could you suggest how to remove the '-' empty lines or fix the replicates found in ledger of TSVB?

Thanks,
Mehak

[flash1293 (Joe Reuter)]

In both Discover and TSVB you should be able to use a filter to filter those documents out:
the KQL query yourFieldName : *filters down to the documents that have the field yourFieldName set. I'm not sure whether the field is actually empty in your case or whether it just contains a - symbol. In the latter case you can filter it out using NOT yourFieldName: '-'

In TSVB you can set the filter in the Options tab of the series:

[Mehak_Bhargava]

Thanks @flash1293, this helped!