Visualization update with new fields


(M.A. Monjas) #1

Hi all,

I'm trying to update a visualization with new fields as they arrive to Elasticsearch. Let me explain:

  • I have a complete ELK system with Elasticsearch, Kibana and Logstash. I'm sending information from a Spark Streaming cluster throuh Kafka. The information is JSON-encoded and is properly stored in the Elasticsearch index.
  • Examples of the JSON messages are as follows:
    {"Details": [{"label-03": 944}, {"label-08": 803}, {"label-05": 218}]}
    {"Details": [{"label-07": 398}, {"label-09": 797}, {"label-06": 895}, {"label-04": 744}, {"label-02": 608}]}
    {"Details": [{"label-02": 235}, {"label-03": 889}, {"label-06": 483}, {"label-04": 67}, {"label-04": 194}]}
  • I'm trying to create a bar plot with the average amount of label-nn. The key issue here is that I don't know beforehand the names of the fields. Of course, everything has to be automatic, with no need to manually refresh anything.

In short, I'd like to have a visualization where as new fields are being added to the ES index, the visualization shows them.

I've made some unsuccessful attempts but to no avail:

  • I've created a visualization with available fields (index previously loaded in Kibana). The visualization is fine.
  • I've created a Python script to reload the fields of the indices (it seems to work):

[code]import calendar, time
import requests

config = {
    'hostname': '10.65.104.181',
    'port': '5601'
}

url = 'http://%s:%s/elasticsearch/*/_mapping/field/*' %(config['hostname'], config['port'])
values = {'_': str(calendar.timegm(time.gmtime())*1000),
          'ignore_unavailable': 'false',
          'allow_no_indices': 'false',
          'include_defaults': 'true'}

r = requests.get(url, params=values)[/code]
  • Next, I've updated the visualization by means of the Import visualization URL:
config = {
        'hostname': '10.65.104.181',
        'port': '5601'
    }
new_item = {u'type': u'avg',
            u'enabled': True,
            u'id': None,
            u'schema': u'metric',
            u'params': {u'field': u''}
           }
visualization_id = 'fe8e8300-9a16-11e7-aab5-e99b4e59c080'

url_download = 'http://%s:%s/es_admin/.kibana/_mget'
body_download ='{"docs":[{"_id":"%s","_type":"visualization"}]}' %(config['hostname'], config['port'], visualization_id)
headers_download = {"content-type": "application/json; charset=UTF-8",
                    "kbn-xsrf": "Visualization download"
                    }

url_upload = u'http://%s:%s/es_admin/.kibana/visualization/%s' %(config['hostname'], config['port'], visualization_id)
headers_upload = {"Accept": "application/json, text/plain, */*",
                  "content-type": "application/json",
                  "kbn-xsrf": "Visualization upload"
                  }

r = requests.post(url_download, headers=headers_download, data=body_download)
visualization = json.loads(r.text)["docs"][0][u"_source"]

_item = deepcopy(new_item)
   # An example
_item[u'id'] = u'10'
_item[u'params'][u'field'] = u'Details.label-08'

items = json.loads(visualization[u"visState"])
items["aggs"].append(_item)

coded_items = json.dumps(items)
visualization[u"visState"] = coded_items

body_upload = json.dumps(visualization)
r = requests.post(url_upload, headers=headers_upload, data=body_upload)[/code]

The procedure seems to work, as the new fields are stored in the visualization. However, although the visualization has a 5 second refresh timer (and the values of existing fields are updated), no new bar appears. When I go to the visualization menu and access to the visualization, the new bar is there, but the visualization does not get updated.

Is there any other workaround for this or definitely it's a feature not supported?

Best regards and many thanks into advance

// M.A. Monjas


(Lee Drengenberg) #2

You can use your browser's Dev console to see the request that Kibana sends to Elasticsearch when you click the Refresh button on the index pattern. For example,

https://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=dlstest&meta_fields=["_source"%2C"_id"%2C"_type"%2C"_index"%2C"_score"]

When URL decoded is a little easier to read;
https://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=dlstest&meta_fields=["_source","_id","_type","_index","_score"]

I think if you did that, and then updated the index pattern doc in .kibana with those results, that might do it...

Note that what I posted above is from Kibana 5.6.1 and things like this change between some releases (and may change in the future, breaking your code).

I don't think I've heard of another user asking for an API to be able to refresh an index pattern before, but you could file an enhancement request in the Kibana repo https://github.com/elastic/kibana/issues to ask for it.

Regards,
Lee


(Lee Drengenberg) #3

Actually, look at (and maybe comment on);

and/or;


Reg kibana UI - unable to view default index
(M.A. Monjas) #4

Thanks, I'll try to proceed as suggested and see whether it works...

Best regards

// M.A.


(M.A. Monjas) #5

Well, the mapping refresh seems to work. However, what definitely does not seem to work is the visualization update. In the same way as suggested, I used the browser developer tool to determine what was under the hood. First I tried with the visualization upload option. First I downloaded the JSON visualization description, updated it and uploaded it, with no result. Next I tried with a POST to_msearch, as it seems to be how a new field is added to a visualization in the visualization menu. Although I get a positive response, the visualization does not get updated :frowning:

Maybe I'm using a wrong approach and tweaking the input is a more convenient way. In short:

  • I'm reporting the number of connections associated to a given service, identified by a label. The name and number of services is not known beforehand.
  • I want to have a visualization (bar, pie, not actually relevant) showing the top services according to accummulated number of connections (or average, not actually relevant).
  • I want such a visualization to get automatically updated.

Is it possible?

Best regards

// M.A.


(Jim Unger) #6

Hi!

Can you post your mapping here?

Also, can you post the elasticsearch request for your visualization?


(system) #7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.