# Failed to decode ndjson new format with python

**URL:** <https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881>\
**Category:** Kibana\
**Created:** [October 9, 2019, 4:58pm UTC](https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881 "2019-10-09T16:58:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![GitsBdr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gitsbdr/32/58828_2.png) [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\
**Post date:** [October 9, 2019, 4:58pm UTC](https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881/1 "2019-10-09T16:58:42Z")

</div>

Hi here,  
Thanks in advance for your time and help

I can find on the web couple of examples using python scripts to "decode" data from the Kibana API with json  
Such as :

```
dashboard = requests.get('http://' + hostname + ':' + port + '/.kibana/dashboard/'+ sys.argv[1] + '?pretty')
json_dashboard = dashboard.json()

```

With the new format ndjson, I adapted my code as below :

```
import ndjson
import requests

response = requests.post('https://example.com/api/data')
items = response.json(cls=ndjson.Decoder)

```

And it works fine in most of the cases !  
But for few visualizations I'm getting :  
`Traceback (most recent call last): File "./script", line 181, in <module> main(clientId, menuId) File "./script", line 146, in main showIds(clientInfo["ClientKibanaPort"],user_choice) File "./script", line 72, in showIds items = response.json(cls=ndjson.Decoder) File "/usr/lib/python2.7/site-packages/requests/models.py", line 818, in json self.content.decode(encoding), **kwargs File "/usr/lib64/python2.7/site-packages/simplejson/ __init__.py", line 533, in loads return cls(encoding=encoding,**kw).decode(s) File "/usr/lib/python2.7/site-packages/ndjson/codecs.py", line 8, in decode text = '[{}]'.format(lines) UnicodeEncodeError: 'ascii' codec can't encode characters in position 44598-44606: ordinal not in range(128)`

I've tried to convert my 'response' to utf8 like that :

```
text = ndjson.dumps()

```

But afterwards I've got :

`Traceback (most recent call last): File "./script", line 188, in <module> main(clientId, menuId) File "./script", line 153, in main showIds(clientInfo["ClientKibanaPort"],user_choice) File "./script", line 73, in showIds text = ndjson.dumps(response) File "/usr/lib/python2.7/site-packages/ndjson/api.py", line 26, in dumps return json.dumps(*args, **kwargs) File "/usr/lib64/python2.7/json/ __init__.py", line 250, in dumps sort_keys=sort_keys,**kw).encode(obj) File "/usr/lib/python2.7/site-packages/ndjson/codecs.py", line 16, in encode line = super(Encoder, self).encode(each, *args, **kwargs) File "/usr/lib64/python2.7/json/encoder.py", line 201, in encode return encode_basestring_ascii(o) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc2 in position 127: unexpected end of data`

I know isn't completely related to Kibana but other users might experience this kind of issue I presume

Thanks  
Guillaume

---

<div class="post-metadata">

**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\
**Post date:** [October 9, 2019, 10:47pm UTC](https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881/2 "2019-10-09T22:47:57Z")

</div>

Thanks for sharing this here, Guillaume! Have you reported this bug to the `ndjson` package maintainers? Can we rule out that this is a bug with that package? If you can help me understand that this is a bug on our side and not `ndjson` then I'd like to know what you think is causing this error?

---

<div class="post-metadata">

**Author:** ![GitsBdr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gitsbdr/32/58828_2.png) [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)\
**Post date:** [October 10, 2019, 6:00am UTC](https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881/3 "2019-10-10T06:00:51Z")

</div>

Hi @cjcenizal

Been working on it yesterday - I actually found out at least on my problems

Can't really figure out what happened but in few visualizations it appears that I had "­­­­­­­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­" in a custom label. (it doesn't seem to be spaces in a row but another character ...)

Well, once the export done, the custom label looks like that :

```
{
  \"enabled\":true,
  \"id\":\"6\",
  \"params\":{
    \"customLabel\":\"­­­­­­­­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­\",
    \"field\":\"YYYY_MM\",
    \"missingBucket\":false,
    \"missingBucketLabel\":\"Missing\",
    \"order\":\"desc\",
    \"orderBy\":\"1\",
    \"otherBucket\":false,
    \"otherBucketLabel\":\"Other\",
    \"row\":false,
    \"size\":12
  },
  \"schema\":\"split\",
  \"type\":\"terms\"
}

```

Which seems to be converted by spaces too by the browser but it actually looks like :

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/a/ca8e75f2385744027bafca4908e21b202002ebd0.png)  
on my NotePad

and like :

![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e21c7b34f0aff6c506274749ddcc21596f99cc91.png)

on my output console

Would you have any idea of what is this ?

Regards  
Guillaume

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 7, 2019, 6:06am UTC](https://discuss.elastic.co/t/failed-to-decode-ndjson-new-format-with-python/202881/4 "2019-11-07T06:06:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.