# Bulk Insert using helpers

**URL:** <https://discuss.elastic.co/t/bulk-insert-using-helpers/126836>\
**Category:** Elasticsearch\
**Created:** [April 5, 2018, 4:16am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836 "2018-04-05T04:16:43Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sunil\_Kalyanpur](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@Sunil\_Kalyanpur](https://discuss.elastic.co/u/Sunil_Kalyanpur)\
**Post date:** [April 5, 2018, 4:16am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836/1 "2018-04-05T04:16:43Z")

</div>

I'm using Elasticsearch 6.2. [This](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-type-field.html) link mentions that `_type` would be removed from future versions.

Hence, I tried to do a bulk insert without specifying `_type` in `_make_documents` function. But, it threw below error.

```
elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: type is missing;')

```

Below is the code snippet used.

```
 def _generate_id(record, unique_id_fields):
      """
    :param record: Document
    :param unique_id_fields: List of fields used to build the Document ID
    :return: Document Id
    """
    return "-".join([str(record[u]) for u in unique_id_fields])

def _make_documents(records, index, type, unique_field=None, unique_id_fields=None, op_type="index"):
    """

    :param records: List of documents to be added to index
    :param index: The name of the index
    :param unique_id_fields: List of fields used to build the Document ID
    :param op_type: If `create` then ensure that the document doesn't exist yet in your index otherwise the call will fail.
                    `index` will add or replace a document as necessary
    :return:
    """
   
    for record in records:
        doc = {
            '_op_type': op_type,
            '_index': index,
            '_type': type,
            '_id': record[unique_field] if unique_field else _generate_id(record=record, unique_id_fields=unique_id_fields),
            '_source': record
        }
        yield (doc)

def bulk_insert(es_config, records, index, type, unique_id_fields, op_type="index", chunk_size=100, refresh=True, aws=True):
    """

    :param es_config: ES connection details
    :param records: Records to insert
    :param index: The name of the index
    :param unique_id_fields: List of fields to build the Document Id
    :param op_type: `create` or `index` or `update`
    :param chunk_size
    :param refresh
    :param aws: ES location `aws` vs `localhost`
    :return:
    """
    es = get_es_conn(config=es_config, aws=aws)
    return helpers.bulk(client=es,
                        actions=_make_documents(records=records, index=index, type=type, unique_id_fields=unique_id_fields, op_type=op_type),
                        chunk_size=chunk_size,
                        refresh=refresh)

```

Is `_type` same as mapping type? Is it OK to hardcode `_type` value to `document` within an index? Please advise.

Sample Mapping type example below:

```
{
  "mappings": {
    "doc": {
      "properties": {
        "type": { "type": "keyword" }, 
        "name": { "type": "text" },
        "user_name": { "type": "keyword" },
        "email": { "type": "keyword" },
        "content": { "type": "text" },
        "tweeted_at": { "type": "date" }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [April 5, 2018, 4:47am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836/2 "2018-04-05T04:47:28Z")

</div>

The (mapping) type is going to disappear, yet it is still there and you need to specify it.  
The common consensus is to use `_doc` or `doc` as the type, similar to what you have in your sample mapping at the bottom.

---

<div class="post-metadata">

**Author:** ![Sunil\_Kalyanpur](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@Sunil\_Kalyanpur](https://discuss.elastic.co/u/Sunil_Kalyanpur)\
**Post date:** [April 5, 2018, 5:11am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836/3 "2018-04-05T05:11:44Z")

</div>

Thank you for the response. Could you please provide more clarity on below items:

1. `The (mapping) type is going to disappear, yet it is still there and you need to specify it.`
2. If I understood your response, `_type` is different from `mapping type`. Is that right?

---

<div class="post-metadata">

**Author:** ![val](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/val/32/138203_2.png) [@val](https://discuss.elastic.co/u/val)\
**Post date:** [April 5, 2018, 6:31am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836/4 "2018-04-05T06:31:09Z")

</div>

1. I invite you to read this: [https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch](https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch)
2. `_type` and mapping type are the same thing

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 3, 2018, 6:31am UTC](https://discuss.elastic.co/t/bulk-insert-using-helpers/126836/5 "2018-05-03T06:31:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.