Here's the situation. A source shipping logs to our cluster has messed up and sent us a large number of documents containing undesirable fields. The thing is that the fields names are almost random, there is no way to list them all. There is a common pattern to all of them but also each of them differ in some way.
I am searching for a way to reindex the documents so I can remove said fields. However so far all the ways I have found imply that the name of the fields to be removed must be known. Is there any way to maybe create a pipeline that could isolate the fields using a regular expression or any other way I am not seeing?
Any suggestion you might have will be very welcome.
Have you tried Ingest Pipeline with script processors? It could be a solution. Regular expression is supported in painless.
I've raised Remove all fields other than a defined list · Issue #83010 · elastic/elasticsearch · GitHub to see if we can add an option to define a list of fields to keep, and then remove all the others, for the remove ingest processor.
It doesn't help you now sorry!
Or if you can list up all the fields you need and set mappings for them, set
dynamic: false on new index could be an option.
That is precisely what I am trying to do. However what I am not seeing is how to isolate the fields to be removed without specifically naming each one by name and removing them. What I am not finding is how to, say, loop on them by using a regular expression since there are all different in name save for one common pattern that repeats itself in their name.
Thank you! Much appreciated!
I cannot list them, which is actually the exact problem with my situation.
Oh wait. I see what you mean here. Sorry my currently uncafeinated stated caused me to misread you post.
Will explore this option for sure.
Well that worked. Hopefully a more flexible solution will exist in a not-too-far future but your suggestion worked for the time being. Thank you good sir!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.