Hi all,
I'm trying to update a visualization with new fields as they arrive to Elasticsearch. Let me explain:
- I have a complete ELK system with Elasticsearch, Kibana and Logstash. I'm sending information from a Spark Streaming cluster throuh Kafka. The information is JSON-encoded and is properly stored in the Elasticsearch index.
- Examples of the JSON messages are as follows:
{"Details": [{"label-03": 944}, {"label-08": 803}, {"label-05": 218}]}
{"Details": [{"label-07": 398}, {"label-09": 797}, {"label-06": 895}, {"label-04": 744}, {"label-02": 608}]}
{"Details": [{"label-02": 235}, {"label-03": 889}, {"label-06": 483}, {"label-04": 67}, {"label-04": 194}]}- I'm trying to create a bar plot with the average amount of label-nn. The key issue here is that I don't know beforehand the names of the fields. Of course, everything has to be automatic, with no need to manually refresh anything.
In short, I'd like to have a visualization where as new fields are being added to the ES index, the visualization shows them.
I've made some unsuccessful attempts but to no avail:
- I've created a visualization with available fields (index previously loaded in Kibana). The visualization is fine.
- I've created a Python script to reload the fields of the indices (it seems to work):
[code]import calendar, time
import requests
config = {
'hostname': '10.65.104.181',
'port': '5601'
}
url = 'http://%s:%s/elasticsearch/*/_mapping/field/*' %(config['hostname'], config['port'])
values = {'_': str(calendar.timegm(time.gmtime())*1000),
'ignore_unavailable': 'false',
'allow_no_indices': 'false',
'include_defaults': 'true'}
r = requests.get(url, params=values)[/code]
- Next, I've updated the visualization by means of the Import visualization URL:
config = {
'hostname': '10.65.104.181',
'port': '5601'
}new_item = {u'type': u'avg',
u'enabled': True,
u'id': None,
u'schema': u'metric',
u'params': {u'field': u''}
}visualization_id = 'fe8e8300-9a16-11e7-aab5-e99b4e59c080'
url_download = 'http://%s:%s/es_admin/.kibana/_mget'
body_download ='{"docs":[{"_id":"%s","_type":"visualization"}]}' %(config['hostname'], config['port'], visualization_id)
headers_download = {"content-type": "application/json; charset=UTF-8",
"kbn-xsrf": "Visualization download"
}
url_upload = u'http://%s:%s/es_admin/.kibana/visualization/%s' %(config['hostname'], config['port'], visualization_id)
headers_upload = {"Accept": "application/json, text/plain, */*",
"content-type": "application/json",
"kbn-xsrf": "Visualization upload"
}
r = requests.post(url_download, headers=headers_download, data=body_download)
visualization = json.loads(r.text)["docs"][0][u"_source"]
_item = deepcopy(new_item)
# An example
_item[u'id'] = u'10'
_item[u'params'][u'field'] = u'Details.label-08'
items = json.loads(visualization[u"visState"])
items["aggs"].append(_item)
coded_items = json.dumps(items)
visualization[u"visState"] = coded_items
body_upload = json.dumps(visualization)
r = requests.post(url_upload, headers=headers_upload, data=body_upload)[/code]
The procedure seems to work, as the new fields are stored in the visualization. However, although the visualization has a 5 second refresh timer (and the values of existing fields are updated), no new bar appears. When I go to the visualization menu and access to the visualization, the new bar is there, but the visualization does not get updated.
Is there any other workaround for this or definitely it's a feature not supported?
Best regards and many thanks into advance
// M.A. Monjas
