Different dashboard from multiple indexes for different project logs

[DG13]

Hi,
Use case : Multiple project logs file
Kibana : Each project log files is stored as one index like project1 index is index1 , project2 index project2

I am new to ELK stack , Need advise do I need to create multiple dashboards for each project index or I can use the same dashboard and can configure the index id in dashboard through API call ?
If anybody has faced this kind of use case and have better solution let me know .

Thanks ,
Divya

[flash1293 (Joe Reuter)]

Hi @DG13,

It is possible to have a single dashboard with visualizations using different indices as data source as the index pattern is always tied to the individual visualization, not the dashboard, and you can place any visualization on a dashboard.

The steps are as follows:

• Go to Management > Index patterns and configure one index pattern per project index. And index pattern is a set of indices but it can also contain just a single one (you've probably already done this)
• Go to Visualize and create your visualizations . If you create a new visualization, you will be prompted to select an index pattern as data source. A best practice is to prefix the name with something like [project1] to make it obvious what index pattern the visualization is referring to.
• Go to Dashboard, create a new dashboard and add all the visualizations you want, no matter the index pattern.

However there is one pitfall: If the indices used on the same dashboard contain different fields (e.g. index1 contains the field fieldA but not fieldB and index2 doesn't contain fieldA but does contain fieldB), you have to be careful while applying filters to the dashboard because they will be applied to all the visualizations regardless the underlying index. This means if you filter by fieldB to equal value1, all visualizations using index1 will stop showing data because there won't be a single document in index1 mathcing this criteria. If your indices all feature the same fields, there won't be a problem.

[DG13]

You mean I have to create the visualization per project index that may be lead me to duplicate the visualizations . As I have 3 dashboards currently and each dashboard has 7 visualizations
If I follow the technique you suggested I have to duplicate 37 visualizations and If I have 10 projects then it will 103*7 .
Correct me if my understanding is wrong

Thanks,
Divya

[flash1293 (Joe Reuter)]

Thats correct, but if you have that many projects and you want to have identical dashboards for each of them there is a simpler way.

• Ingest the data with an additional field project that holds the value project1, project2 and so on.
• Create an index pattern that covers all of your project indices, e.g. if your indices are called index1, index2, ..., then your index pattern should be index*.
• Create your visualizations once referencing the "all projects" index pattern.
• Create a single dashboard and add all visualizations
• Place a filter on the dashboard filtering down on the project field to a single project. Now you can switch between the different project views by changing the filter value on a single dashboard.

This has the additional advantage of not having to change n dashboards if you want to change something on all of them. If you want to "branch out" a dashboard for a specific project and make a separate dashboard, you can also clone your dashboard and save it under a new name.

The process outlined above makes sense if you want to have different views depending on the project.

[DG13]

That's good idea but caveat here is, if the index grows in my case 10 project with daily refresh . Doesn't that affect the performance ?
My thoughts on solution ( unsure about the feasibility of the solution has anybody ever tried it ? )

1. Create landing page having link for each project for exampleProject1
2. Place the index id as variable in all the visualizations
3. When user clicks on the link It should pass the index id to the visualization and display that project specific data (index) to the dashboards

Is this solution possible ?
Is there any way we can pass the index id to the visualization ?

Thanks,
Divya

[flash1293 (Joe Reuter)]

The index pattern is always tied to the visualization, it's not possible to parameterize it without changing the saved object.

One way to do that would be to use the Saved Objects API to programmatically copy visualizations and dashboard while changing the index pattern.

But I would try to keep it simple in the beginning - Elasticsearch is quite performant regarding these things and does a lot of optimizations under the hood. Once it becomes a problem it's still possible to craft a more sophisticated solution.