Bulk Insert using helpers

I'm using Elasticsearch 6.2. This link mentions that _type would be removed from future versions.

Hence, I tried to do a bulk insert without specifying _type in _make_documents function. But, it threw below error.

elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: type is missing;')

Below is the code snippet used.

 def _generate_id(record, unique_id_fields):
      """
    :param record: Document
    :param unique_id_fields: List of fields used to build the Document ID
    :return: Document Id
    """
    return "-".join([str(record[u]) for u in unique_id_fields])


def _make_documents(records, index, type, unique_field=None, unique_id_fields=None, op_type="index"):
    """

    :param records: List of documents to be added to index
    :param index: The name of the index
    :param unique_id_fields: List of fields used to build the Document ID
    :param op_type: If `create` then ensure that the document doesn't exist yet in your index otherwise the call will fail.
                    `index` will add or replace a document as necessary
    :return:
    """
   
    for record in records:
        doc = {
            '_op_type': op_type,
            '_index': index,
            '_type': type,
            '_id': record[unique_field] if unique_field else _generate_id(record=record, unique_id_fields=unique_id_fields),
            '_source': record
        }
        yield (doc)


def bulk_insert(es_config, records, index, type, unique_id_fields, op_type="index", chunk_size=100, refresh=True, aws=True):
    """

    :param es_config: ES connection details
    :param records: Records to insert
    :param index: The name of the index
    :param unique_id_fields: List of fields to build the Document Id
    :param op_type: `create` or `index` or `update`
    :param chunk_size
    :param refresh
    :param aws: ES location `aws` vs `localhost`
    :return:
    """
    es = get_es_conn(config=es_config, aws=aws)
    return helpers.bulk(client=es,
                        actions=_make_documents(records=records, index=index, type=type, unique_id_fields=unique_id_fields, op_type=op_type),
                        chunk_size=chunk_size,
                        refresh=refresh)

Is _type same as mapping type? Is it OK to hardcode _type value to document within an index? Please advise.

Sample Mapping type example below:

{
  "mappings": {
    "doc": {
      "properties": {
        "type": { "type": "keyword" }, 
        "name": { "type": "text" },
        "user_name": { "type": "keyword" },
        "email": { "type": "keyword" },
        "content": { "type": "text" },
        "tweeted_at": { "type": "date" }
      }
    }
  }
}

The (mapping) type is going to disappear, yet it is still there and you need to specify it.
The common consensus is to use _doc or doc as the type, similar to what you have in your sample mapping at the bottom.

Thank you for the response. Could you please provide more clarity on below items:

  1. The (mapping) type is going to disappear, yet it is still there and you need to specify it.
  2. If I understood your response, _type is different from mapping type. Is that right?
  1. I invite you to read this: https://www.elastic.co/blog/removal-of-mapping-types-elasticsearch
  2. _type and mapping type are the same thing

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.