# Field with different types on same index

**URL:** <https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047>\
**Category:** Elasticsearch\
**Created:** [September 23, 2022, 6:33pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047 "2022-09-23T18:33:38Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![prabello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabello/32/111183_2.png) [@prabello](https://discuss.elastic.co/u/prabello)\
**Post date:** [September 23, 2022, 6:33pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/1 "2022-09-23T18:33:38Z")

</div>

Currently, we are indexing information on elastic for each business that is our customer (B2B), this is causing us to have too many indexes, some with very small shards (500mb shards)

But due to the nature of the data, we can have a lot of type collision on the fields, for example  
Company\_A, has the following:

```auto
{
    "id": "8f22efb2-6a2a-4cb7-9d0b-01ca0d6cff2e",
    "notes" : {
        "created_at": "2022/01/01",
        "value": "This is some random note"
    } ,
    "first_name": "Some name",
    "integration": "CRM_A"
}

```

While Company\_B will have:

```auto
{
    "id": 1,
    "notes": "This is a random note text",
    "name": "Some random name",
    "integration": "CRM_A"
}

```

What we were thinking is to have them on a single index, by "integration", but on this case, they would both have different types for the field called `notes`, where one has a object, another one has text, this also happens on a lot of other fields, basically multiple cases of the same field name with different types

Is there any way to solve this with elastic?  
If we can have a way, we would finally be able to have our shards with 20-30gb of data, as is recommended, instead of having TONS of small shards, forcing us to have way more memory to keep all the mappings and everything in place.

One thing we considered was to add the type to the name of the fields, and remove them prior to returning the response, but that might takes really confusing 😕

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 23, 2022, 9:54pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/2 "2022-09-23T21:54:32Z")

</div>

Short answer is those two schemas cannot live in a single index the way that you have them.

As you know in one schema `notes` is an object and another schema `notes` is just a concrete field.

The way to solve this is choose one of the schemas to standardize to and use an ingest pipeline to standardize the fields.

In this case, it's probably easier to just standardize the concrete into the `notes.value` field.

---

<div class="post-metadata">

**Author:** ![prabello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabello/32/111183_2.png) [@prabello](https://discuss.elastic.co/u/prabello)\
**Post date:** [September 23, 2022, 10:41pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/3 "2022-09-23T22:41:26Z")

</div>

My biggest issue is that happens with lots of different fields (some of the documents have more than 30 collisions :/)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 23, 2022, 10:45pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/4 "2022-09-23T22:45:39Z")

</div>

Hmmmm that _is_ unfortunate... no real simple answer there...

Unless you put each set under a top level Field... then you wont have collisions...

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 23, 2022, 11:21pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/5 "2022-09-23T23:21:33Z")

</div>

Just in case you are interest here is a sample ingest pipeline / simulate with various combinations

```auto
## Ingest Pipeline Concrete / Object
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "rename": {
          "if": "ctx?.notes instanceof String", 
          "field": "notes",
          "target_field": "notes.value",
          "ignore_failure": false
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "id": "8f22efb2-6a2a-4cb7-9d0b-01ca0d6cff2e",
        "notes": {
          "created_at": "2022/01/01",
          "value": "This is some random note"
        },
        "first_name": "Some name",
        "integration": "CRM_A"
      }
    },
    {
      "_source": {
        "id": 1,
        "notes": "This is a random note text",
        "name": "Some random name",
        "integration": "CRM_A"
      }
    },
    {
      "_source": {
        "id": 1,
        "name": "Some random name",
        "integration": "CRM_A"
      }
    }
  ]
}

```

Results

```auto
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "integration": "CRM_A",
          "notes": {
            "created_at": "2022/01/01",
            "value": "This is some random note"
          },
          "id": "8f22efb2-6a2a-4cb7-9d0b-01ca0d6cff2e",
          "first_name": "Some name"
        },
        "_ingest": {
          "timestamp": "2022-09-23T23:23:13.461958276Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "name": "Some random name",
          "integration": "CRM_A",
          "id": 1,
          "notes": {
            "value": "This is a random note text"
          }
        },
        "_ingest": {
          "timestamp": "2022-09-23T23:23:13.461985402Z"
        }
      }
    },
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "name": "Some random name",
          "integration": "CRM_A",
          "id": 1
        },
        "_ingest": {
          "timestamp": "2022-09-23T23:23:13.461991898Z"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![prabello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prabello/32/111183_2.png) [@prabello](https://discuss.elastic.co/u/prabello)\
**Post date:** [September 23, 2022, 11:41pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/6 "2022-09-23T23:41:33Z")

</div>

Wow, thanks a lot stephenb! Gonna take a stab at it

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 21, 2022, 11:42pm UTC](https://discuss.elastic.co/t/field-with-different-types-on-same-index/315047/7 "2022-10-21T23:42:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.