Extract the Key from MQTT JSON message

Elastic Stack Beats
3

Hello!

If I understand correctly, you would like to insert one document in ES for each of the {"n":"","u":"","v":""} objects contained inside the measures array, am I correct? So from your example

{"device_info": {"uuid": "XXXXXX","fw_ver": "V1.5.0" },"measures": [{"n":"co2", "u":"ppm", "v":1155.995},{"n":"voc", "u":"ppm", "v":0.000e-3},{"n":"co", "u":"ppm", "v":3.898},{"n":"pm10", "u":"ug/m3", "v":3.713},{"n":"pm2.5", "u":"ug/m3", "v":3.713},{"n":"temp", "u":"Cel", "v":22.515},{"n":"hum", "u":"%RH", "v":63.681},{"n":"prb", "u":"hPa", "v":1007.199},{"n":"pm1", "u":"ug/m3", "v":3.511},{"n":"pm4", "u":"ug/m3", "v":3.713},{"n":"iaqi", "u":"count", "v":66},{"n":"tci", "u":"count", "v":84},{"n":"eiaqi", "u":"count", "v":5}]}

in ES you expect the following documents:

{"n":"co2", "u":"ppm", "v":1155.995}

{"n":"voc", "u":"ppm", "v":0.000e-3}

{"n":"co", "u":"ppm", "v":3.898}

{"n":"pm10", "u":"ug/m3", "v":3.713}

{"n":"pm2.5", "u":"ug/m3", "v":3.713}

{"n":"temp", "u":"Cel", "v":22.515}

{"n":"hum", "u":"%RH", "v":63.681}

{"n":"prb", "u":"hPa", "v":1007.199}

{"n":"pm1", "u":"ug/m3", "v":3.511}

{"n":"pm4", "u":"ug/m3", "v":3.713}

{"n":"iaqi", "u":"count", "v":66}

{"n":"tci", "u":"count", "v":84}

{"n":"eiaqi", "u":"count", "v":5}

If that is the case, this is not currently supported by filebeat or ingest processors, as they expect 1 event in and 1 event out, and you would need 1 event in and N events out. One solution could be to change the MQTT messages format to only contain one measurement in each of them, and then just ingesting them separately. Alternatively, it is possible to use script or decode_json_message processors to flatten the list and have all measurements at the document root, which would help a lot while querying or using them in dashboards. As a last resort, if it is an option and you can switch MQTT for HTTP/S, then the httpjson input is capable of creating one document for each of them using its split configuration.

LMK if this helped in any way! :slight_smile: