JSON Payload to Elastic.py

[Matthew_Butler]

Here is my JSON payload:

{"device":"368640","deviceType":"Sigfox-Tracker","version":3,"time":"Tue Aug 21 2018 02:33:05 GMT+0000 (UTC)","rssi":"-120.00","battery":3.28,"temp":20.87,"soil":1915,"xAxis":59,"yAxis":-38,"zAxis":994}

I'm using the script https://github.com/mattfield11/mqtt-elasticSearch/blob/master/mqttToElasticSearch.py to import json payloads from MQTT into elasticsearch, however it imports the entire payload as a string in the index and therefore I can't search for the fields correctly. I'm new to elasticsearch, could anyone help me correct the code slightly?

[pjanzen (Paul Janzen)]

Hi,

In the project from mattfield11 I cannot find anything on how he is using the data in his project(s).
Depending on which version of elasticsearch you are using you have several options.

1. You setup logstash, create a filter that decompose the data and pushes it to logstash. Start with and setup logstash
2. You setup a pipeline in elasticsearch (I have never done this so I cannot help) pipeline docs.

Other than that I cannot help you much further.

Good luck,
Paul.

[pjanzen (Paul Janzen)]

Browsing the internet and researching the mqtt thing from mattfield11 I found this link. Now as this is not something I want to know I did not read any further but it might help you in setup and understanding this mqtt stuff.

Also he has a per-requisite thingy right here.

[Matthew_Butler]

I had ran into another problem of .kibana becoming read only, but I've fixed that now!

Basically, the top data payload is the one provided by the script, whilst the bottom payload is something that I've manually put in (for mapping purposes). How could I edit his script that it would reproduce something like that?

[pjanzen (Paul Janzen)]

Matthew_Butler,

I would be wise to read the links I posted and follow what he created. Again I don't know what mqtt is nor do I want know in that respect I cannot help you.

For completelyness (is that even a word):
1. link one
1. link two

[Matthew_Butler]

Thank you Paul, those were the initial tutorials that I followed to set up the ELK and MQTT server. I've contacted the owner of the github to see if he has any additions, as I'm sure importing could be solved through his script.

I've completed the project. The code required to mount the payload was
es.index(index='landslide', doc_type='mapping', body=msg.payload)

Just in case anyone else wants to use it.