Kibana 5 - best practice for multiple environments

[tallavi]

Hi,

I have dev and testing environments, I pointed both filebeats to the same Kibana instance, with different index prefixes, of course.

It looks like the best practice for me, instead of managing different elastic search and Kibana instances.

BUT, when I create a visualization, I seem to be needed to choose the index, and that visualization only fits that index, and not the other, so I'm ending up with making double visualizations and double dashboards and it's really confusing.

Another idea I had is having both environments in the same index, with a field denoting the environment, that way I can filter to one environment, but I'm not sure I'm feeling comfortable with merging data from different environments. Seems messier that having each environment in it's own index.

What is the best practice here? I would have expected to be able to change the index directly from the dashboard and have it influence on all the shown visualizations, just like I change the date range.

Thanks in advance,

Tal

[timroes]

Hi,

sorry for the very late reply.

It is not possible to quickly switch the index that backs all visualizations on a dashboard, since basically every visualization could be backed by a different index pattern, filters and search.

From where to store your data, I would agree and not store it within the same index. You might even not want to store it within the same cluster.

In general I would recommend using separate indexes for those data with the same mapping. You could (and I guess it would be wise) to add a field containing the actual environment, even if stored in several index patterns.

If you name your index patterns mylogs-dev-* and mylogs-test-* you could also add a third index pattern to Kibana mylogs-*. That way you can chose for any visualization whether you just want to show dev or testing data or data from both environments.

Also when using the mylogs-* index pattern you can now easily create a filter using the "environment" field mentioned above to toggle whether you just want to see data from dev or testing, but still be able to keep that data within different indexes.

You could also store it in completely different clusters if you feel like, and use Cross Cluster Search to query over multiple clusters for your data.

Cheers,
Tim

[tallavi]

Hi @timroes,

Thanks for the reply!

Wow, that's nice! I thought that the * part has to just be the date.

So Kibana should be able to ignore the 'dev- ' and the 'test-' part and just take the date at the end without any changes?

Thanks,

Tal

[timroes]

Yeah the wildcard doesn't have to match a date, it will really just match anything

And yeah it will just read now data from all indexes. That's why you should make sure those indexes still have the same mapping, otherwise you will get a mapping collision.