Programmatically rename (re-title) Kibana index patterns

[GrahamHannington]

I've developed a set of Kibana dashboards that use about a dozen index patterns with the following structure:

product_x-table_name-*

where product_x is a fixed string and table_name has a dozen or so values. Example index pattern title (well, not a real example, but close enough):

product_x-cputimes-*

(I'm using Kibana 7.11.2, about to upgrade to 7.13.x.)

All good so far.

Problem: Product X has multiple components. Each component has its own set of tables. It turns out that the table names are not unique across all components. Components A and B might both have a table named cputimes, but with different sets of columns (fields). For various reasons—such as avoiding data type mapping conflicts, restricting the field selection list in Kibana to the table from a specific component—it makes sense to introduce a component identifier as a qualifier in the Elasticsearch index names and the corresponding Kibana index patterns.

I'm not concerned about the existing Elasticsearch indices. These are currently ephemeral, for development use. I can delete them and start ingesting into new indices from scratch using a Logstash config that specifies an index name containing a component qualifier.

I am concerned about the Kibana index patterns. I don't want to have to, for example, manually create new index patterns, and then update my visualizations (over 100 of them) one at a time. That is unpalatable to me.

Instead, I want to programmatically change the index pattern titles, and all references to those index patterns. One detail that makes this easier: the current index patterns—in my primary dev environment, which is all I'm concerned about here—are all for the same component. So, given a text file that contains the current index pattern titles, I could use simple regex search'n'replaces to update the titles.

I have a specific plan in mind for how to do this. But I thought I'd post this topic first to get feedback and suggestions from users who have already been here, or from the Elastic developers themselves.

Two options occur to me:

1. Write a script that uses the Kibana find objects API to iterate over saved objects, and then use the update object API to update them
2. Export all saved objects to a .ndjson file, use an editor to perform the regex search'n'replace, and then import the updated file, overwriting the existing objects.

Option 2 sounds easier to me. Especially since, based on my inspection of the .ndjson file, it appears as if some of the updates I need to make are inside long string values: JSON "embedded" as an escaped string value of an "actual" JSON key (visState).

I've inspected a .ndjson file of exported saved objects. I can see the lines that define each index pattern. I can see that, in many cases, other objects refer to the index pattern id rather than its title. However, I can also see instances where other objects cite the index pattern title.

Here is a complete list of string patterns I've identified that I need to update:

• The index pattern saved object itself:
  • "title":"product_x-table_name-*"
• Other saved objects:
  • \"series_index_pattern\":\"product_x-table_name-*\"
  • \"default_index_pattern\":\"product_x-table_name-*\"
  • \"index_pattern\":\"product_x-table_name-*\"

For the first string pattern, I could use the following (Java) regex search:

(\"title\":\"product_x-)([^-]+-\*\")

and replace:

$1comp_a-$2

The other updates are also straightforward.

Have I missed anything?

[GrahamHannington]

I donned my chaps, deleted all my indices, ingested data into new product_x-comp_a-* indices, opened an export.ndjson file in a text editor, applied the following Java global regex search:

(\"product_x-)([^-]+-\*)

and replace:

$1comp_a-$2

Then I imported the edited file, overwriting the old saved objects, and...

It worked.

I've done ad hoc inspections of my dashboards and seen the newly titled index patterns in the saved objects list and in the Discover tab. No apparent issues.

A single regex search'n'replace appears to have done the trick.

Before I do this in my real dev environment, rather than a sandbox: experienced users, Elastic devs: am I kidding myself?

[flash1293 (Joe Reuter)]

Your approach seems good to me - however make sure to keep the original export file so you can roll back when things go wrong

A bit of context:
The index pattern saved object (the one with the title attribute) is used by some types of visualizations, but other visualizations (namely TSVB, Timelion and Vega) do not use them - that's why they store the pattern string in their own saved objects (series_index_pattern et al).

In 7.13 TSVB will be able to reference the index pattern saved object instead of storing the pattern string inline, simplifying this task a bit.

I think your approach catches Lens, aggregation based visualizations and TSVB. Vega and Timelion are more difficult to migrate this way because they put the pattern string into their respective non-JSON based "specs" which would have to be parsed separately for a robust replacement:

For reference: I described two other approaches of changing the index pattern with visualizations attached here: Rename the index-pattern for sample visualizations They don't require importing/exporting but won't catch the TSVB cases, so I think your scheme is better in your case.

[GrahamHannington]

Hi @flash1293 , thanks very much for the detailed info, much appreciated. Yeah, no Timelion or Vega yet in this set of dashboards. And, yes, I'll be able to roll back .