Bulk Import of data for different indices in presence of enrich ingest pipeline

Elastic Stack Elasticsearch
1

Context:

  • I have two indices: devices and events
  • devices has an enrich policy associated with it and a pipeline was created to enable this data enrichment. This pipeline works fine.
  • i want to bulk import JSON objects for devices and events. When i bulk import data for both indices, the pipeline does not work.
  • when i bulk import data for devices only, the pipeline works.

Does anyone know why this happens and is there a work around for this?

2

It'd be useful if you could share an example of the bulk request you are sending.

1 Like
3

Sorry about that. Here is the bulk request:

POST /_bulk
{"update":{"_index":"mydevices","_id":"7f:74:47:8e:aa:2c"}}
{"doc":{"adv_addr":{"value":"7f:74:47:8e:aa:2c","type":"public"},"manufacturer":{"id":"0x0075","data":"QgQBgGZUOtZ1HllWOtZ1HlgBAAAAAAAA"},"flag":["general_discoverable","classic_not_supported"],"service_list":{"complete":false,"services":[{"uuid":"00001811-0000-1000-8000-00805F9B34FB"}]}},"doc_as_upsert":true}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"7f:74:47:8e:aa:2c"},"timestamp":1648516430788,"rssi":-89}
{"update":{"_index":"mydevices","_id":"41:7b:81:3c:29:4e"}}
{"doc":{"adv_addr":{"value":"41:7b:81:3c:29:4e","type":"random"},"flag":["general_discoverable","dual_mode_controller","dual_mode_host"],"manufacturer":{"id":"0x004c","data":"EAc4H4LYuF5o"}},"doc_as_upsert":true}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"7f:74:47:8e:aa:2c"},"timestamp":1648516430846,"rssi":-86}
{"update":{"_index":"mydevices","_id":"41:7b:81:3c:29:4e"}}
{"doc":{"adv_addr":{"value":"41:7b:81:3c:29:4e","type":"random"}},"doc_as_upsert":true}
{"update":{"_index":"mydevices","_id":"7f:74:47:8e:aa:3c"}}
{"doc":{"adv_addr":{"value":"7f:74:47:8e:aa:3c","type":"public"},"flag":["general_discoverable","classic_not_supported"],"service_list":{"complete":false,"services":[{"uuid":"0000FE59-0000-1000-8000-00805F9B34FB"}]}},"doc_as_upsert":true}
{"update":{"_index":"mydevices","_id":"59:1e:75:d6:3a:54"}}
{"doc":{"adv_addr":{"value":"59:1e:75:d6:3a:54","type":"public"},"manufacturer":{"id":"0x0075","data":"QgQBgGZUOtZ1HllWOtZ1HlgBAAAAAAAA"}},"doc_as_upsert":true}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"7f:74:47:8e:aa:2c"},"timestamp":1648516430788,"rssi":-89}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"7f:74:47:8e:aa:2c"},"timestamp":1648516430846,"rssi":-86}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"7f:74:47:8e:aa:2c"},"timestamp":1648516430903,"rssi":-89}
{"index":{"_index":"theevents"}}
{"adv_addr":{"value":"59:1e:75:d6:3a:54"},"timestamp":1648516430807,"rssi":-86}

The pipeline enriches the manufacturer.id and services.uuid fields.

I just tested it and the pipeline only works if i bulk import the same data twice. Is there a way for me to accomplish the enrichment by bulk importing once?

4

It also works fine if i do a simple curl request

5

Where are you defining the pipeline, is it just a default one for the index/indices?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.