# Cluster keeps becoming unresponsive

**URL:** <https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677>\
**Category:** Elasticsearch\
**Created:** [November 26, 2015, 2:13pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677 "2015-11-26T14:13:22Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 2:13pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/1 "2015-11-26T14:13:22Z")

</div>

Hey,

Our complete three node cluster keeps on becoming completely unresponsive. The index concerned is up and running on one node, but isn't able to replicate to the other two nodes due to a mapping issue, apparently.

This is the issue

`Mapper for [swiped_price] conflicts with existing mapping in other types[object mapping [swiped_price] can't be changed from non-nested to nested]`

I understand what it's saying, but how do I resolve the issue?

Please help!

Ro.

---

<div class="post-metadata">

**Author:** ![GlenRSmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenrsmith/32/44925_2.png) [@GlenRSmith](https://discuss.elastic.co/u/GlenRSmith)\
**Post date:** [November 26, 2015, 3:28pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/2 "2015-11-26T15:28:12Z")

</div>

Yoltsro,

The Found team pinged me (and others) on this - Having a look now, may ask for more info in a few.

---

<div class="post-metadata">

**Author:** ![GlenRSmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenrsmith/32/44925_2.png) [@GlenRSmith](https://discuss.elastic.co/u/GlenRSmith)\
**Post date:** [November 26, 2015, 3:46pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/3 "2015-11-26T15:46:27Z")

</div>

Ro,

I hope you can bear with me. Can you help me understand your workflow better?

How/when did you create this index? What version of ES did you create it in? Did you upgrade your ES version afterward?

Do you have multiple types defined for the index?

Can provide the output of a call to GET /\_mappings?

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 3:50pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/4 "2015-11-26T15:50:48Z")

</div>

We created this index around 1 week ago, I think. We created it in ES 2.0. We haven't upgraded our version of ES.

We have two types defined in the index: products and interactions:

[https://gist.github.com/yoitsro/feb0fccddf10ac18c2a5](https://gist.github.com/yoitsro/feb0fccddf10ac18c2a5)

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:07pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/5 "2015-11-26T16:07:42Z")

</div>

Hi @yoitsro

You have two fields called "swiped\_price", one is of type "object" and the other is of type "nested". This kind of mapping conflict is no longer allowed in 2.0. Fields with the same name should have the same mapping, so you need to change one of those fields. I'm guessing that "swiped\_price" can contain arrays of objects, in which case "nested" is more than likely the one you want to use.

See [https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html) for more on nested fields.

You should also delete the "\_id" mapping - it is no longer configurable (although you're using the default "string" type and it will be accepted).

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:11pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/6 "2015-11-26T16:11:05Z")

</div>

Hi Clinton,

Thanks for that. That `swiped_price` field on the products type was an accidental insert of a document which automatically created that field. However, I can now no longer delete it. The only other option is to reindex our data, but with our current situation as it is is (seemingly random failures), I can't do this. I've already reindexed the majority of the products collection into the mallzee\_v4 index, but am very nervous about reindexing the interactions documents.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:16pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/7 "2015-11-26T16:16:09Z")

</div>

@yoitsro Are you saying that you created the swiped\_price mapping in the interaction type with type nested, then indexed a document into "products" and it introduced the new "swiped\_price" object mapping dynamically?

Do you have any dynamic mapping rules?

---

<div class="post-metadata">

**Author:** ![GlenRSmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenrsmith/32/44925_2.png) [@GlenRSmith](https://discuss.elastic.co/u/GlenRSmith)\
**Post date:** [November 26, 2015, 4:16pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/8 "2015-11-26T16:16:57Z")

</div>

Ro,

What version of ES was the \_v3 index created under?

I'd like to get to the bottom of how \_v3 got to its current state, because, as Clinton explains, it contains mappings which violate constraints present in ESv2.0+.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:20pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/9 "2015-11-26T16:20:07Z")

</div>

I've just tried creating this index on 1.7.0, but it throws the same exception when I try to upgrade to 2.0 (as it should)

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:22pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/10 "2015-11-26T16:22:04Z")

</div>

That is what happened. I don't have any dynamic mapping rules as of yet, but will need to add them once we can get to the bottom of this.

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:23pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/11 "2015-11-26T16:23:29Z")

</div>

The \_v3 index was created in ESv2.0. Like I say, the `swiped_price` was added dynamically.

---

<div class="post-metadata">

**Author:** ![GlenRSmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenrsmith/32/44925_2.png) [@GlenRSmith](https://discuss.elastic.co/u/GlenRSmith)\
**Post date:** [November 26, 2015, 4:27pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/12 "2015-11-26T16:27:01Z")

</div>

swiped\_price was added dynamically to \_v3 products, correct? It's defined as nested on \_v3 interactions, and nested mappings don't happen dynamically (unless maybe you have an index template. Do you?)

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:30pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/13 "2015-11-26T16:30:57Z")

</div>

OK something is very weird:

- The fields in `products.swiped_price.*` have custom mappings, not the ones that would be added by dynamic mapping without any custom rules.
- You say that the `swiped_price` field was added dynamically by indexing a document into the index, but it also exists in the \_v4 index (which has the same custom mappings). Do you have a product document that contains `swiped_price` that you reindexed?

I've tried following the steps you describe, but I get the expected exception... Any more details that you can provide to help me recreate this?

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:31pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/14 "2015-11-26T16:31:03Z")

</div>

Correct - swiped\_price was added dynamically to \_v3 products. We don't have an index template.

Because of the error, I deleted the document containing the swiped\_price field and tried to update the mapping on products to be nested. ES replied with acknowledged.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:35pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/15 "2015-11-26T16:35:40Z")

</div>

I'm going to need more details of what happened when, so I can try to recreate this.

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:38pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/16 "2015-11-26T16:38:44Z")

</div>

Yeah, just mentioned this in the other reply - I did try to add the field after I saw the errors.

In the \_v4 index, I've update the mapping so that swiped\_price is no longer nested to prevent against potential future issues. I will also turn off dynamic mapping on \_v4 shortly.

In order to reproduce, try the following:

Create the \_v3 index with the attached mapping, BUT remove the swiped\_price field from the products mapping.

Insert a document into products matching that format:

```auto
{
    "name": "Aces Plain Overshirt",
    "pid": "acesplainovershirt",
    "type": "shirts",
    "sku": "1041551812",
    "description": "Workwear with a twist! This season, we’ve updated our versatile artisan overshirt. Unlined and crafted from a great herringbone cotton for an authentic utility feel, it also features two useful hand pockets. \n\nUtility Workwear Overshirt\nConstructed from a durable herringbone cotton",
    "variant_link_id": "9b7823f16fe81e91031f711a104fb688",
    "images": [
      "http://cdn.shopify.com/s/files/1/0302/5117/products/RE0353P_Navy_1.jpg?v=1440145870",
      "http://cdn.shopify.com/s/files/1/0302/5117/products/RE0353P_Navy_3.jpg?v=1440145870",
      "http://cdn.shopify.com/s/files/1/0302/5117/products/RE0353P_Navy_2.jpg?v=1440145870"
    ],
    "brand": {
      "_id": "5579a4ddaf95c4cd2f93c12d",
      "name": "Realm and Empire"
    },
    "parent_id": "55dcff6a8856831600188f61",
    "primary_category": "outerwear",
    "uniform_sort_time": "2015-08-24T15:35:20.260Z",
    "buy_url": "http://www.realmandempire.com/collections/outerwear/products/aces-plain-overshirt-navy-re0353p",
    "enabled": true,
    "sizes": [
      {
        "_id": "3982334532",
        "size": "x small",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      },
      {
        "_id": "3982334596",
        "size": "small",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      },
      {
        "_id": "3982334660",
        "size": "medium",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      },
      {
        "_id": "3982334724",
        "size": "large",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      },
      {
        "_id": "3982334788",
        "size": "x large",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      },
      {
        "_id": "3982334852",
        "size": "2x large",
        "quantity": 15,
        "price": {
          "currency": "GBP",
          "current": 120,
          "on_sale": false
        }
      }
    ],
    "created_at": "2015-08-25T23:47:23.456Z",
    "variants": [],
    "interactions": [],
    "suggest": [
      "shirts",
      "Realm and Empire",
      "Realm & Empire",
      "Realm and Empire shirts",
      "navy shirts",
      "navy Realm and Empire shirts"
    ],
    "product_url": "http://www.realmandempire.com/collections/outerwear/products/aces-plain-overshirt-navy-re0353p",
    "region": "GBR",
    "colour": {
      "hex": "#51769f",
      "name": "navy",
      "original": "navy"
    },
    "gender": "male",
    "merchant": {
      "_id": "5579a24f8b22b92caa95fa2c",
      "name": "Realm & Empire"
    },
    "updated_at": "2015-11-21T10:23:43.188Z",
    "stock_id": "navy"
  }

```

Then update the document so it just contains the sizes (this happened accidentally for a few products).

Then update the document again so it contains the swiped\_price field:

```auto
"swiped_price": {
    "currency": "GBP",
    "current": 5.99
}

```

This is how I got to this state.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:47pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/17 "2015-11-26T16:47:30Z")

</div>

Got it, that's a nasty bug! I'll open an issue to get that fixed, thanks for the recreation.

(Btw, I think you've done the wrong thing in v4... you WANT swiped\_price to be `nested`, not `object`)

Question is how to get you out of this state...

How many shards do you have? What are you seeing in the logs?

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:48pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/18 "2015-11-26T16:48:42Z")

</div>

The cluster's just gone down again 😢 Is there any way to get the cluster responsive again easily?

We have 6 shards and the logs keep complaining about the mapping conflict.

---

<div class="post-metadata">

**Author:** ![yoitsro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yoitsro/32/6192_2.png) [@yoitsro](https://discuss.elastic.co/u/yoitsro)\
**Post date:** [November 26, 2015, 4:51pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/19 "2015-11-26T16:51:51Z")

</div>

I'll change the mapping and reindex, assuming the cluster won't fall over again. Is there another way we can do this which isn't likely to take down the cluster?

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [November 26, 2015, 4:55pm UTC](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677/20 "2015-11-26T16:55:15Z")

</div>

Honestly, I don't know how to get you out of this state quickly without deleting the index.

How many docs are we talking about? Few enough to be able to reindex before the cluster is likely to fall over again? If not, either delete the index and lose the data or consider starting up a new cluster and reindexing the data bit by bit as you can...

Can you paste what you're seeing in the logs? Not just the exception we already know about. I'm looking for why the cluster goes down.

[Next page](https://discuss.elastic.co/t/cluster-keeps-becoming-unresponsive/35677.md?page=2)