# REINDEX : identify fields to remove with a regular expression? **URL:** \ **Category:** Elasticsearch\ **Tags:** reindex\ **Created:** [January 24, 2022, 7:34pm UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275 "2022-01-24T19:34:28Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [January 24, 2022, 7:34pm UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/1 "2022-01-24T19:34:28Z")
Hello. Here's the situation. A source shipping logs to our cluster has messed up and sent us a large number of documents containing undesirable fields. The thing is that the fields names are almost random, there is no way to list them all. There is a common pattern to all of them but also each of them differ in some way. I am searching for a way to reindex the documents so I can remove said fields. However so far all the ways I have found imply that the name of the fields to be removed must be known. Is there any way to maybe create a pipeline that could isolate the fields using a regular expression or any other way I am not seeing? Any suggestion you might have will be very welcome. ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [January 25, 2022, 1:00am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/2 "2022-01-25T01:00:05Z")
Have you tried Ingest Pipeline with script processors? It could be a solution. Regular expression is supported in painless. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 25, 2022, 1:17am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/3 "2022-01-25T01:17:07Z")
I've raised [Remove all fields other than a defined list · Issue #83010 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/83010) to see if we can add an option to define a list of fields to keep, and then remove all the others, for the [remove ingest processor](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/remove-processor.html). It doesn't help you now sorry! ---
**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\ **Post date:** [January 25, 2022, 1:43am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/4 "2022-01-25T01:43:56Z")
Or if you can list up all the fields you need and set mappings for them, set `dynamic: false` on new index could be an option. ---
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [January 25, 2022, 11:55am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/5 "2022-01-25T11:55:23Z")
That is precisely what I am trying to do. However what I am not seeing is how to isolate the fields to be removed without specifically naming each one by name and removing them. What I am not finding is how to, say, loop on them by using a regular expression since there are all different in name save for one common pattern that repeats itself in their name. ---
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [January 25, 2022, 11:56am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/6 "2022-01-25T11:56:36Z")
Thank you! Much appreciated! ---
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [January 25, 2022, 11:57am UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/7 "2022-01-25T11:57:29Z")
I cannot list them, which is actually the exact problem with my situation. ---
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [January 25, 2022, 12:14pm UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/8 "2022-01-25T12:14:49Z")
Oh wait. I see what you mean here. Sorry my currently uncafeinated stated caused me to misread you post. 😅 Will explore this option for sure. ---
**Author:** ![TheEvilDonut](https://avatars.discourse-cdn.com/v4/letter/t/7cd45c/32.png) [@TheEvilDonut](https://discuss.elastic.co/u/TheEvilDonut)\ **Post date:** [February 1, 2022, 1:03pm UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/9 "2022-02-01T13:03:24Z")
Well that worked. Hopefully a more flexible solution will exist in a not-too-far future but your suggestion worked for the time being. Thank you good sir! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 1, 2022, 1:04pm UTC](https://discuss.elastic.co/t/reindex-identify-fields-to-remove-with-a-regular-expression/295275/10 "2022-03-01T13:04:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.