Create-or-replace index programmatically

Elastic Stack Elasticsearch
1

Hi,

I have a JAVA web service that uses an ES index for caching. Sometimes the field/index definitions change so the ES index must be re-created.

What I am looking for is a solution pattern (or API function) that:

  1. Creates index and mapping if it is missing
  2. Deletes and re-creates index and mapping if the index definition is not in sync with an expected definition
2

Creates index and mapping if it is missing

I wrote a project which does something like this:

You can see it in this demo project. Index settings.

Deletes and re-creates index and mapping if the index definition is not in sync with an expected definition

That's super dangerous because it will delete all the existing data.
But there is an option in Beyonder here:

Using force will remove any existing index if exists. Not that it will not compare the existing schema with the one that you want to apply. That could be a good option to add to this project though. But it is still super dangerous to me and I'd only use that for integration testing not for production code.

Note that the merge option can try to update an existing mapping but it will work only in a few circonstances (like adding a new field).

3

Thanks for the hint! - But I am really looking for something robust that detects index definition changes and performs a complete re-creation if definitions have changed. Data loss is not an issue - ES is not the primary data store and can be rebuilt within minutes. It is just not convenient to force that kind of re-creation on every service restart.

4

You need to write yourself this logic then.
I'm not sure if spring data elasticsearch or hibernate search support that but this is something you might want to check.

If you do your own implementation, I'd be happy to welcome it as a PR in the beyonder project. :wink:

5

I see three options how information about the last "valid" index definition could be persisted:

  1. Hash the original index definition and store hash it in some metadata field in ES (I am not sure there is an appropriate field to do this - is there?)
  2. Hash the original index definition and store hash it in a persistent location (usually File System)
  3. Keep the original index definition and store it in a persistent location (also FS)

On startup, the current _settings.json could be checked against the persisted hash/definition and force re-creation if they don't match.

My preference would be (1) but I could not yet find an appropriate customizable metadata field at the index level. Options (2,3) require configuration of some persistent file location, which would work for me but is probably less desirable for this kind of library.

6

Good idea. I'd use https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-meta-field.html

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.