# Errors reindexing documents from version 6 to version 7

**URL:** <https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970>\
**Category:** Elasticsearch\
**Tags:** reindex\
**Created:** [November 29, 2020, 12:16am UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970 "2020-11-29T00:16:57Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [November 29, 2020, 12:16am UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/1 "2020-11-29T00:16:57Z")

</div>

I have a several hundred daily indexes indexed under versions 5 and 6. I have recently upgraded to 7.10 and now want to clean up and standardise my index mappings.

I am attempting to reindex using a ruby program and the gem : elasticsearch-extensions-0.0.31

```auto
  def do_reindex( name = nil, new_name = nil)
    name ||= @name
    new_name ||= @options[:new_name]
    reindex = Elasticsearch::Extensions::Reindex.new source: { index: name, client: @client }, target: { index: new_name }
    reindex.perform
end

```

this results in output to stderr:

```
warning: 299 Elasticsearch-7.10.0-51e9d6f22758d0374a0f3f5c6e8f3a7997850f96 "[types removal] Specifying types in bulk requests is deprecated."

```

the reindex perform returns: `{:errors=>120853}`, i.e one for each doc in the original index.

and in the cluster logs we see:

> [2020-11-29T12:31:47,239][INFO][o.e.a.b.TransportShardBulkAction] [secesprd02] [authentication\_2019.09.02][1] mapping update rejected by primary  
> java.lang.IllegalArgumentException: mapper [event\_source] cannot be changed from type [keyword] to [text]  
> at org.elasticsearch.index.mapper.ParametrizedFieldMapper.merge(ParametrizedFieldMapper.java:90) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.ParametrizedFieldMapper.merge(ParametrizedFieldMapper.java:61) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.ObjectMapper.doMerge(ObjectMapper.java:535) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.RootObjectMapper.doMerge(RootObjectMapper.java:298) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.ObjectMapper.merge(ObjectMapper.java:495) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.RootObjectMapper.merge(RootObjectMapper.java:293) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.Mapping.merge(Mapping.java:110) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.DocumentMapper.merge(DocumentMapper.java:281) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:470) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:421) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:361) ~[elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:292) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:179) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) [elasticsearch-7.10.0.jar:7.10.0]  
> at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.0.jar:7.10.0]  
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]  
> at java.lang.Thread.run(Thread.java:832) [?:?]

since upgrading to 7.10 the mapping of the source index shows:

```auto
{
  "authx_2019.09.02": {
    "mappings": {
      "properties": {

```

i.e. no type

and I don't give any type in the reindex.

What am I doing wrong?

---

<div class="post-metadata">

**Author:** ![egalpin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egalpin/32/79772_2.png) [@egalpin](https://discuss.elastic.co/u/egalpin)\
**Post date:** [November 29, 2020, 3:16am UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/2 "2020-11-29T03:16:53Z")

</div>

> [@Russell\_Fulton](#):
>
> elasticsearch-extensions-0.0.31

It looks like this client is out of date. The client is still using scroll + bulk API to emulate reindex API ([source](https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-extensions/lib/elasticsearch/extensions/reindex.rb#L92)). I believe scroll+bulk pre-dates the official reindex API.

In the source code of the extension you mentioned, the scroll results are taken verbatim and mirrored to bulk requests. I’m sure the `_type` field is present in the hits from the scroll query, and that’s where the `_type` makes its way into the bulk requests and generates the errors.

I’d recommend looking for a more up-to-date client, and/or looking over the [Reindex API Docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [November 29, 2020, 3:43pm UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/3 "2020-11-29T15:43:16Z")

</div>

> [@Russell\_Fulton](#):
>
> ```auto
> "[types removal] Specifying types in bulk requests is deprecated."
> 
> ```

This is indeed an indication that your client is using an older, deprecated, form of bulk request, but it's not the problem here. It will become a problem in the next major version so you need to address it before you upgrade to 8.x, but you can ignore it for now.

The actual problem is included in the bulk indexing responses. It doesn't look like this library makes it easy to see this, but I think client-side [transport logging](https://github.com/elastic/elasticsearch-ruby/tree/master/elasticsearch-transport#logging) will show you the details. I lack the Ruby expertise to tell you exactly how to get this, sorry.

The log you shared does suggest the problem might be related to mappings, specifically some confusion around the type of the `event_source` field. It's trying to guess the type of this field dynamically, which means you're not setting up the mappings before starting the reindex process. I recommend doing that.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [November 29, 2020, 3:44pm UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/4 "2020-11-29T15:44:51Z")

</div>

> [@DavidTurner](#):
>
> you need to address it before you upgrade to 8.x

Eh I just realised this library is developed by Elastic, so don't worry we'll get to this task before 8.x is released.

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [November 29, 2020, 7:18pm UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/5 "2020-11-29T19:18:14Z")

</div>

Ah, I missed the mention of event\_source in the transport logs. Doh! I am fundamentally not visually orientated and I find picking out vital information from the screeds of java traceback really difficult unless I know what I am looking for.

Turns out the problem is really simple! A typo in the index pattern for the target template!

Thanks for the help folks and I will work on the ruby side. I suspect that I am picking up the original reindex call (which will have been left for backward compatibility) and not the one in the extension.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [November 29, 2020, 8:21pm UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/6 "2020-11-29T20:21:01Z")

</div>

> [@Russell\_Fulton](#):
>
> I find picking out vital information from the screeds of java traceback really difficult unless I know what I am looking for.

That's why we're here 🙂 There is a bit of a knack to it...

> [@Russell\_Fulton](#):
>
> Thanks for the help folks and I will work on the ruby side. I suspect that I am picking up the original reindex call (which will have been left for backward compatibility) and not the one in the extension.

The one in the extension (i.e. [this one](https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-extensions/lib/elasticsearch/extensions/reindex.rb)) is doing a client-side reindex (i.e. scroll + bulk indexing). Nothing wrong with that, but if you want the server-side reindex it's [this one](https://github.com/elastic/elasticsearch-ruby/blob/master/elasticsearch-api/lib/elasticsearch/api/actions/reindex.rb) which is in the main `elasticsearch-api` library instead.

The server-side one does basically the same thing (scroll + bulk indexing) just without involving the client. I don't think it would have worked in this case either, for the same reason.

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [November 29, 2020, 11:32pm UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/7 "2020-11-29T23:32:06Z")

</div>

I now have the reindexing picking up the correct template but I am now getting errors for every document: `mapper [level] cannot be changed from type [integer] to [text]` but the mapping of both the original and the target indexes have :

```auto
 "level": {
          "type": "integer"
  }

```

when I look at the index properties using cerebro.

Weird! Any idea how I have stuffed it up this time?

BTW this happens with both implementations of reindex! My program now has a switch to tell it which call to use.

---

<div class="post-metadata">

**Author:** ![Russell\_Fulton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/russell_fulton/32/62888_2.png) [@Russell\_Fulton](https://discuss.elastic.co/u/Russell_Fulton)\
**Post date:** [November 30, 2020, 3:12am UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/8 "2020-11-30T03:12:25Z")

</div>

> I get it! reindex is working off the \_source where level is in fact a text string. I used a script to  
> explicitly cast it to int and that worked.

editing since this may have been emailed...

No, it worked because a small change to the template json caused ES to ignore the "properties" completely.

So back to trying to figure out why it thinks there is a miss match in the field types.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2020, 3:12am UTC](https://discuss.elastic.co/t/errors-reindexing-documents-from-version-6-to-version-7/256970/9 "2020-12-28T03:12:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.