REINDEX : identify fields to remove with a regular expression?

Elastic Stack Elasticsearch
1

Hello.

Here's the situation. A source shipping logs to our cluster has messed up and sent us a large number of documents containing undesirable fields. The thing is that the fields names are almost random, there is no way to list them all. There is a common pattern to all of them but also each of them differ in some way.

I am searching for a way to reindex the documents so I can remove said fields. However so far all the ways I have found imply that the name of the fields to be removed must be known. Is there any way to maybe create a pipeline that could isolate the fields using a regular expression or any other way I am not seeing?

Any suggestion you might have will be very welcome.

2

Have you tried Ingest Pipeline with script processors? It could be a solution. Regular expression is supported in painless.

3

I've raised Remove all fields other than a defined list · Issue #83010 · elastic/elasticsearch · GitHub to see if we can add an option to define a list of fields to keep, and then remove all the others, for the remove ingest processor.

It doesn't help you now sorry!

4

Or if you can list up all the fields you need and set mappings for them, set dynamic: false on new index could be an option.

5

That is precisely what I am trying to do. However what I am not seeing is how to isolate the fields to be removed without specifically naming each one by name and removing them. What I am not finding is how to, say, loop on them by using a regular expression since there are all different in name save for one common pattern that repeats itself in their name.

6

Thank you! Much appreciated!

7

I cannot list them, which is actually the exact problem with my situation.

8

Oh wait. I see what you mean here. Sorry my currently uncafeinated stated caused me to misread you post. :sweat_smile:

Will explore this option for sure.

9

Well that worked. Hopefully a more flexible solution will exist in a not-too-far future but your suggestion worked for the time being. Thank you good sir!