# Visualization update with new fields

**URL:** <https://discuss.elastic.co/t/visualization-update-with-new-fields/100881>\
**Category:** Kibana\
**Created:** [September 18, 2017, 2:12pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881 "2017-09-18T14:12:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![mmonjas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmonjas/32/22214_2.png) [@mmonjas](https://discuss.elastic.co/u/mmonjas)\
**Post date:** [September 18, 2017, 2:12pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/1 "2017-09-18T14:12:02Z")

</div>

Hi all,

I'm trying to update a visualization with new fields as they arrive to Elasticsearch. Let me explain:

- I have a complete ELK system with Elasticsearch, Kibana and Logstash. I'm sending information from a Spark Streaming cluster throuh Kafka. The information is JSON-encoded and is properly stored in the Elasticsearch index.
- Examples of the JSON messages are as follows:

```auto
    {"Details": [{"label-03": 944}, {"label-08": 803}, {"label-05": 218}]}
    {"Details": [{"label-07": 398}, {"label-09": 797}, {"label-06": 895}, {"label-04": 744}, {"label-02": 608}]}
    {"Details": [{"label-02": 235}, {"label-03": 889}, {"label-06": 483}, {"label-04": 67}, {"label-04": 194}]}
```

- I'm trying to create a bar plot with the average amount of label-nn. The key issue here is that I don't know beforehand the names of the fields. Of course, everything has to be automatic, with no need to manually refresh anything.

In short, I'd like to have a visualization where as new fields are being added to the ES index, the visualization shows them.

I've made some unsuccessful attempts but to no avail:

- I've created a visualization with available fields (index previously loaded in Kibana). The visualization is fine.
- I've created a Python script to reload the fields of the indices (it seems to work):

[code]import calendar, time  
import requests

```
config = {
    'hostname': '10.65.104.181',
    'port': '5601'
}

url = 'http://%s:%s/elasticsearch/*/_mapping/field/*' %(config['hostname'], config['port'])
values = {'_': str(calendar.timegm(time.gmtime())*1000),
          'ignore_unavailable': 'false',
          'allow_no_indices': 'false',
          'include_defaults': 'true'}

r = requests.get(url, params=values)[/code]

```

- Next, I've updated the visualization by means of the Import visualization URL:

```auto
config = {
        'hostname': '10.65.104.181',
        'port': '5601'
    }
```

```auto
new_item = {u'type': u'avg',
            u'enabled': True,
            u'id': None,
            u'schema': u'metric',
            u'params': {u'field': u''}
           }
```

```
visualization_id = 'fe8e8300-9a16-11e7-aab5-e99b4e59c080'

url_download = 'http://%s:%s/es_admin/.kibana/_mget'
body_download ='{"docs":[{"_id":"%s","_type":"visualization"}]}' %(config['hostname'], config['port'], visualization_id)
headers_download = {"content-type": "application/json; charset=UTF-8",
                    "kbn-xsrf": "Visualization download"
                    }

url_upload = u'http://%s:%s/es_admin/.kibana/visualization/%s' %(config['hostname'], config['port'], visualization_id)
headers_upload = {"Accept": "application/json, text/plain, */*",
                  "content-type": "application/json",
                  "kbn-xsrf": "Visualization upload"
                  }

r = requests.post(url_download, headers=headers_download, data=body_download)
visualization = json.loads(r.text)["docs"][0][u"_source"]

_item = deepcopy(new_item)
   # An example
_item[u'id'] = u'10'
_item[u'params'][u'field'] = u'Details.label-08'

items = json.loads(visualization[u"visState"])
items["aggs"].append(_item)

coded_items = json.dumps(items)
visualization[u"visState"] = coded_items

body_upload = json.dumps(visualization)
r = requests.post(url_upload, headers=headers_upload, data=body_upload)[/code]

```

The procedure seems to work, as the new fields are stored in the visualization. However, although the visualization has a 5 second refresh timer (and the values of existing fields are updated), no new bar appears. When I go to the visualization menu and access to the visualization, the new bar is there, but the visualization does not get updated.

Is there any other workaround for this or definitely it's a feature not supported?

Best regards and many thanks into advance

// M.A. Monjas

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [September 18, 2017, 8:27pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/2 "2017-09-18T20:27:12Z")

</div>

You can use your browser's Dev console to see the request that Kibana sends to Elasticsearch when you click the Refresh button on the index pattern. For example,

[https://localhost:5601/api/index\_patterns/\_fields\_for\_wildcard?pattern=dlstest&meta\_fields=["\_source"%2C"\_id"%2C"\_type"%2C"\_index"%2C"\_score"]](https://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=dlstest&meta_fields=%5B%22_source%22%2C%22_id%22%2C%22_type%22%2C%22_index%22%2C%22_score%22%5D)

When URL decoded is a little easier to read;  
[https://localhost:5601/api/index\_patterns/\_fields\_for\_wildcard?pattern=dlstest&meta\_fields=["\_source","\_id","\_type","\_index","\_score"]](https://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=dlstest&meta_fields=%5B%22_source%22,%22_id%22,%22_type%22,%22_index%22,%22_score%22%5D)

I think if you did that, and then updated the index pattern doc in .kibana with those results, that might do it...

Note that what I posted above is from Kibana 5.6.1 and things like this change between some releases (and may change in the future, breaking your code).

I don't think I've heard of another user asking for an API to be able to refresh an index pattern before, but you could file an enhancement request in the Kibana repo [https://github.com/elastic/kibana/issues](https://github.com/elastic/kibana/issues) to ask for it.

Regards,  
Lee

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [September 18, 2017, 8:28pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/3 "2017-09-18T20:28:03Z")

</div>

Actually, look at (and maybe comment on);

> <https://github.com/elastic/kibana/issues/3709>

and/or;

> <https://github.com/elastic/kibana/issues/2236>

---

<div class="post-metadata">

**Author:** ![mmonjas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmonjas/32/22214_2.png) [@mmonjas](https://discuss.elastic.co/u/mmonjas)\
**Post date:** [September 19, 2017, 2:12pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/4 "2017-09-19T14:12:26Z")

</div>

Thanks, I'll try to proceed as suggested and see whether it works...

Best regards

// M.A.

---

<div class="post-metadata">

**Author:** ![mmonjas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mmonjas/32/22214_2.png) [@mmonjas](https://discuss.elastic.co/u/mmonjas)\
**Post date:** [September 19, 2017, 3:11pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/5 "2017-09-19T15:11:01Z")

</div>

Well, the mapping refresh seems to work. However, what definitely does not seem to work is the visualization update. In the same way as suggested, I used the browser developer tool to determine what was under the hood. First I tried with the visualization upload option. First I downloaded the JSON visualization description, updated it and uploaded it, with no result. Next I tried with a POST to\_msearch, as it seems to be how a new field is added to a visualization in the visualization menu. Although I get a positive response, the visualization does not get updated ☹

Maybe I'm using a wrong approach and tweaking the input is a more convenient way. In short:

- I'm reporting the number of connections associated to a given service, identified by a label. The name and number of services is not known beforehand.
- I want to have a visualization (bar, pie, not actually relevant) showing the top services according to accummulated number of connections (or average, not actually relevant).
- I want such a visualization to get automatically updated.

Is it possible?

Best regards

// M.A.

---

<div class="post-metadata">

**Author:** ![BigFunger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bigfunger/32/7323_2.png) [@BigFunger](https://discuss.elastic.co/u/BigFunger)\
**Post date:** [September 19, 2017, 4:37pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/6 "2017-09-19T16:37:43Z")

</div>

Hi!

Can you post your mapping here?

Also, can you post the elasticsearch request for your visualization?

 ![temp](https://us1.discourse-cdn.com/elastic/original/3X/a/1/a19d349dcdec87ae51a19dc677712045dbc6db3f.gif)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2017, 4:37pm UTC](https://discuss.elastic.co/t/visualization-update-with-new-fields/100881/7 "2017-10-17T16:37:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.