I'm using Elasticsearch 6.2. This link mentions that _type would be removed from future versions.
Hence, I tried to do a bulk insert without specifying _type in _make_documents function. But, it threw below error.
elasticsearch.exceptions.RequestError: TransportError(400, u'action_request_validation_exception', u'Validation Failed: 1: type is missing;')
Below is the code snippet used.
def _generate_id(record, unique_id_fields):
"""
:param record: Document
:param unique_id_fields: List of fields used to build the Document ID
:return: Document Id
"""
return "-".join([str(record[u]) for u in unique_id_fields])
def _make_documents(records, index, type, unique_field=None, unique_id_fields=None, op_type="index"):
"""
:param records: List of documents to be added to index
:param index: The name of the index
:param unique_id_fields: List of fields used to build the Document ID
:param op_type: If `create` then ensure that the document doesn't exist yet in your index otherwise the call will fail.
`index` will add or replace a document as necessary
:return:
"""
for record in records:
doc = {
'_op_type': op_type,
'_index': index,
'_type': type,
'_id': record[unique_field] if unique_field else _generate_id(record=record, unique_id_fields=unique_id_fields),
'_source': record
}
yield (doc)
def bulk_insert(es_config, records, index, type, unique_id_fields, op_type="index", chunk_size=100, refresh=True, aws=True):
"""
:param es_config: ES connection details
:param records: Records to insert
:param index: The name of the index
:param unique_id_fields: List of fields to build the Document Id
:param op_type: `create` or `index` or `update`
:param chunk_size
:param refresh
:param aws: ES location `aws` vs `localhost`
:return:
"""
es = get_es_conn(config=es_config, aws=aws)
return helpers.bulk(client=es,
actions=_make_documents(records=records, index=index, type=type, unique_id_fields=unique_id_fields, op_type=op_type),
chunk_size=chunk_size,
refresh=refresh)
Is _type same as mapping type? Is it OK to hardcode _type value to document within an index? Please advise.
Sample Mapping type example below:
{
"mappings": {
"doc": {
"properties": {
"type": { "type": "keyword" },
"name": { "type": "text" },
"user_name": { "type": "keyword" },
"email": { "type": "keyword" },
"content": { "type": "text" },
"tweeted_at": { "type": "date" }
}
}
}
}