'New' index patterns are not specific enough

Elastic Stack Kibana
1

hi all,

last couple of weeks we were having a lot of trouble getting the new kibana to not break our elasticsearch cluster.

this morning i realized what the problem might be. we moved from the (now deprecated) index pattern like "[30mhz-com-]YYYY.MM" to the new "30mhz-com-*". this causes problems because we have index names like 30mhz-com-2015.11.20160521 with an alias like 30mhz-com-2015.11.

if we switch back to the old naming patterns it works like a charm. i couldn't find out how to use a naming pattern like this without using the deprecated way. documentation suggests it should be possible, but kibana doesn't recognize/find the indexes, and i can't create the index pattern.

groet,
jurg.

2

There isn't a way to create a wildcard that will match 30mhz-com-2015.11 but not 30mhz-com-2015.11.20160521, so you either have to keep using the unoptimized [30mhz-com-]YYYY.MM format or change your index patterns to have an identifier of some sort before the date.

Maybe you could change your aliases to 30mhz-com-monthly-2015.11 and then use the 30mhz-com-monthly-* wildcard to match those aliases.

But I'm curious what the purpose of the aliases is. With the new optimizations Kibana will only access the indices necessary to fill out the time range using the field stats api, so if these aliases are simply for making querying from kibana easier with the old date-pattern-based index patterns you could just get rid of them completely.

3

re-indexing is not my favorite hobby. each of our indexes is over 40G,
without shards. changing the naming pattern would work, but it means we'd
have to change different components of our system.

the purpose of the aliases is to make working with indexes more convenient.
the naming of the indexes was just something logical. the naming of the
aliases was logstash format.

how long are you going to keep the old index patterns?

4

For the foreseeable future, but you would benefit from querying the underlying indices directly as Kibana can execute a query for the "last 15 minutes" on the single underlying index, rather than consulting all indices with the 30mhz-com-2016.05 alias

5

i re-indexed all my indexes to a pattern like 30mhz.com-2015.05, with aliases like 30mhz-com-2015.05. changing back from the monthly rotating naming pattern (deprecated) to the new 'optimized' naming pattern breaks my cluster again.

we run a cluster of 6 data nodes (16G), 3 master nodes and 2 client nodes (also 16G). 17 indexes, growing from 60m documents (40G) to 90m documents (54G).