Parallel_bulk record format (update, insert, etc.)

Elastic Stack Elasticsearch
1

Hello,

I have been able to get "helpers.parallel_bulk" in Python (v3) to insert new json records using the the format below (array of JSON records):
[{'_id': 'AVyPTQ2d82rj_AUAq9dc', '_type': 'mytype', '_index': 'myindex-2017.06.10', 'starttime': '2017-06-09T23:59:48.670430Z', 'src': '192.168.0.5', 'sport': 4570, etc ... }
{'_id': 'AVyPTQ2d82rj_AUAq9db', '_type': 'mytype', '_index': 'myindex-2017.06.10', 'starttime': '2017-06-09T23:59:48.670430Z', 'src': '192.168.0.10', 'sport': 3463, etc ... }
{'_id': 'AVyPTQ2d82rj_AUAq9da', '_type': 'mytype', '_index': 'myindex-2017.06.10', 'starttime': '2017-06-09T23:59:48.670430Z', 'src': '192.168.0.11', 'sport': 4987, etc ... }
etc.]

The problem is that the format seems quite different than the bulk update documentation. Namely:
[{"index": {"id": "AVyKQdW282rj_AUAqK4", "_type": "mytype", "_index": "myindex-2017.06.09"}}
{'starttime': '2017-06-09T23:59:48.670430Z', 'src': '192.168.0.11', 'sport': 4987, etc ... }
{"index": {"_id": "AVyKRJ1a82rj_AUAqLGR", "_type": "mytype", "_index": "myindex-2017.06.09"}}
{'starttime': '2017-06-09T23:59:48.670430Z', 'src': '192.168.0.11', 'sport': 4987, etc ... }
etc.]

The issue is that I want to be able to update and index records in the same parallel_bulk request. What is the correct JSON format that I should be using to distinguish between an index, create, update, and delete for each record?

Thanks!

2

For those looking for the answer, I was able to find it. There is a _op_type code that needs to be added. (http://elasticsearch-py.readthedocs.io/en/master/helpers.html). From the documentation index is the default, but it will accept index, create, update, detete, etc.

Here is what the record needs to look like:
{'_id': 'AVyUdVY882rj_AUAr36V', '_type': 'mytype', '_index': 'myindex-2017.06.11', '_op_type': 'update', 'doc': {'fieldtoupdate': 'newvalue'}} [ Note: op type, id, type, and index are missing required underscores on the post because the site removes them ]

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.