# Remove nested fields by query and/or by \_reindex

**URL:** <https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432>\
**Category:** Elasticsearch\
**Created:** [June 5, 2019, 5:41pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432 "2019-06-05T17:41:55Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 5, 2019, 5:41pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/1 "2019-06-05T17:41:55Z")

</div>

I currently was using dynamic\_mapping to create .raw fields for all strings but want to remove those in favor of a .keyword field.  
Trying to figure out the proper ctx.\_source syntax for update\_by\_query or \_reindex to remove these fields from all documents in the index.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 5, 2019, 5:55pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/2 "2019-06-05T17:55:38Z")

</div>

What about using reindex API to read the \_source from the old index and send it to the new index which has the new mapping?

---

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 5, 2019, 5:56pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/3 "2019-06-05T17:56:44Z")

</div>

New mapping is still using dynamic but dynamic with keyword fields instead of raw fields. So either way I would need to ctx.\_source.remove(FIELD.raw)

---

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 5, 2019, 6:08pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/4 "2019-06-05T18:08:15Z")

</div>

I've tried a few things like:

```
POST logs-000068.1/_update_by_query?wait_for_completion=false&conflicts=proceed
{

  "script": {
    "source": """ctx._source.remove("raw")""",
    "lang": "painless"
  },
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "raw"
          }
        }
      ]
    }
  }
}

```

and

```
POST logs-000068.1/_update_by_query
{
  "script": {
    "lang": "painless",
    "source": """
      if (ctx._source.keySet().stream().collect(Collectors.toList()).forEach(k".raw" -> ctx._source.remove(k".raw"))
    """
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 5, 2019, 6:20pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/5 "2019-06-05T18:20:33Z")

</div>

> [@Chris\_Denneen](#):
>
> New mapping is still using dynamic but dynamic with keyword fields instead of raw fields. So either way I would need to ctx.\_source.remove(FIELD.raw)

The important thing is how the `_source` looks like.  
IMHO you won't see any `raw` field in the `_source`.  
So you don't need to remove what does not exist.

Could you share a sample document?

---

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 5, 2019, 6:29pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/6 "2019-06-05T18:29:16Z")

</div>

you are correct the raw isn't in the document itself so this should be fine:

```
{
  "_index": "logs-000068.1",
  "_type": "_doc",
  "_id": "ARNv7WkB31eIUe7WLbyH",
  "_version": 1,
  "_seq_no": 44930422,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "msg": "Received Tagging Service Request.",
    "entitled": "company",
    "apiKey": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXx",
    "activity": "TaggingService.Tag",
    "source": [
      "C:\\Tagging\\TaggingService.log",
      "Metadata.Services.Tagging.TaggingService"
    ],
    "sessionid": "be7ff00c08ce48f69428ca34789a60ff",
    "type": "feed_logs",
    "responseType": "application/rdf+xml",
    "apenv": "prod",
    "log_category": "feed_logs",
    "requestId": "c1ff4eca042f4d569a3975d5af4dbbe2",
    "beat": {
      "hostname": "win-dnr3cmptlrv",
      "name": "win-dnr3cmptlrv",
      "version": "5.6.4"
    },
    "host": "win-dnr3cmptlrv",
    "@version": "1",
    "elapsedTime_t": "2911",
    "app": "apms",
    "Status": "Success",
    "hashdata": "7f22d9020fc199f8c0184b8b4b53c2c0",
    "offset": 3414611,
    "level": "INFO",
    "input_type": "log",
    "message": "2019-04-05 12:17:57.4075|INFO|Metadata.Services.Tagging.TaggingService|_sessionid=\"be7ff00c08ce48f69428ca34789a60ff\" _subcategory=\"NONE\" _activity=\"TaggingService.Tag\" requestId=\"c1ff4eca042f4d569a3975d5af4dbbe2\" apiKey=\"XXXXXXXXXXXXXXXXXXXX\" entitled=\"company\" responseType=\"application/rdf+xml\" Status=\"Success\" _msg=\"Received Tagging Service Request.\" elapsedTime_t=\"2911\"",
    "tags": [
      "beats",
      "beats_input_codec_plain_applied",
      "priority_queue",
      "eastern"
    ],
    "site": "api",
    "@timestamp": "2019-04-05T16:17:57.407Z",
    "cacheKeys_test": null,
    "states_built_test": null,
    "subcategory": "NONE",
    "skipped_min_dates_test": null
  }
}
```

---

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 5, 2019, 6:32pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/7 "2019-06-05T18:32:35Z")

</div>

\_reindex gives me issues sometimes, like here I had to rename a few fields:

```
{
  "completed" : true,
  "task" : {
    "node" : "_b8r5Ci0Qxa2UyiZJIFwrQ",
    "id" : 1432658604,
    "type" : "transport",
    "action" : "indices:data/write/reindex",
    "status" : {
      "total" : 269913131,
      "updated" : 1850000,
      "created" : 0,
      "deleted" : 0,
      "batches" : 185,
      "version_conflicts" : 0,
      "noops" : 0,
      "retries" : {
        "bulk" : 0,
        "search" : 0
      },
      "throttled_millis" : 0,
      "requests_per_second" : -1.0,
      "throttled_until_millis" : 0
    },
    "description" : "reindex from [logs-000070] updated with Script{type=inline, lang='painless', idOrCode='ctx._source['cacheKeys_test'] = ctx._source.remove('cacheKeys_t'); ctx._source['states_built_test'] = ctx._source.remove('states_built_t'); ctx._source['skipped_min_dates_test'] = ctx._source.remove('skipped_min_dates_t')', options={}, params={}} to [logs-000070.1][_doc]",
    "start_time_in_millis" : 1559239057119,
    "running_time_in_nanos" : 1152160768166,
    "cancellable" : true,
    "headers" : { }
  },
  "error" : {
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [2638765]"
        }
      },
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [2287697]"
        }
      },
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [1369216]"
        }
      },
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [373423]"
        }
      },
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [190986]"
        }
      },
      {
        "shard" : -1,
        "index" : null,
        "reason" : {
          "type" : "search_context_missing_exception",
          "reason" : "No search context found for id [1369215]"
        }
      }
    ],
    "caused_by" : {
      "type" : "search_context_missing_exception",
      "reason" : "No search context found for id [1369215]"
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 6, 2019, 7:24am UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/8 "2019-06-06T07:24:11Z")

</div>

Anything in elasticsearch logs which could give more details?  
Any chance a node went down or was overloaded while you were doing that?

---

<div class="post-metadata">

**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\
**Post date:** [June 6, 2019, 6:38pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/9 "2019-06-06T18:38:31Z")

</div>

I don't recall seeing anything in the logs but as you see from that task output it was my second attempt at this "updated" vs "created" and it had same issues. So don't think it has anything to do with overloaded but more with something going wrong with the `_reindex` itself. This was on the 3rd index as I did 2 before it with no issues.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 4, 2019, 6:38pm UTC](https://discuss.elastic.co/t/remove-nested-fields-by-query-and-or-by-reindex/184432/10 "2019-07-04T18:38:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.