Failed to decode ndjson new format with python

[GitsBdr (Guillaume Baudringhien)]

Hi here,
Thanks in advance for your time and help

I can find on the web couple of examples using python scripts to "decode" data from the Kibana API with json
Such as :

dashboard = requests.get('http://' + hostname + ':' + port + '/.kibana/dashboard/'+ sys.argv[1] + '?pretty')
json_dashboard = dashboard.json()

With the new format ndjson, I adapted my code as below :

import ndjson
import requests

response = requests.post('https://example.com/api/data')
items = response.json(cls=ndjson.Decoder)

And it works fine in most of the cases !
But for few visualizations I'm getting :
Traceback (most recent call last): File "./script", line 181, in <module> main(clientId, menuId) File "./script", line 146, in main showIds(clientInfo["ClientKibanaPort"],user_choice) File "./script", line 72, in showIds items = response.json(cls=ndjson.Decoder) File "/usr/lib/python2.7/site-packages/requests/models.py", line 818, in json self.content.decode(encoding), **kwargs File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 533, in loads return cls(encoding=encoding, **kw).decode(s) File "/usr/lib/python2.7/site-packages/ndjson/codecs.py", line 8, in decode text = '[{}]'.format(lines) UnicodeEncodeError: 'ascii' codec can't encode characters in position 44598-44606: ordinal not in range(128)

I've tried to convert my 'response' to utf8 like that :

text = ndjson.dumps()

But afterwards I've got :

Traceback (most recent call last): File "./script", line 188, in <module> main(clientId, menuId) File "./script", line 153, in main showIds(clientInfo["ClientKibanaPort"],user_choice) File "./script", line 73, in showIds text = ndjson.dumps(response) File "/usr/lib/python2.7/site-packages/ndjson/api.py", line 26, in dumps return json.dumps(*args, **kwargs) File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps sort_keys=sort_keys, **kw).encode(obj) File "/usr/lib/python2.7/site-packages/ndjson/codecs.py", line 16, in encode line = super(Encoder, self).encode(each, *args, **kwargs) File "/usr/lib64/python2.7/json/encoder.py", line 201, in encode return encode_basestring_ascii(o) UnicodeDecodeError: 'utf8' codec can't decode byte 0xc2 in position 127: unexpected end of data

I know isn't completely related to Kibana but other users might experience this kind of issue I presume

Thanks
Guillaume

[cjcenizal]

Thanks for sharing this here, Guillaume! Have you reported this bug to the ndjson package maintainers? Can we rule out that this is a bug with that package? If you can help me understand that this is a bug on our side and not ndjson then I'd like to know what you think is causing this error?

[GitsBdr (Guillaume Baudringhien)]

Hi @cjcenizal

Been working on it yesterday - I actually found out at least on my problems

Can't really figure out what happened but in few visualizations it appears that I had "­­­­­­­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­ ­" in a custom label. (it doesn't seem to be spaces in a row but another character ...)

Well, once the export done, the custom label looks like that :

{
  \"enabled\":true,
  \"id\":\"6\",
  \"params\":{
    \"customLabel\":\"­­­­­­­­ ­  ­  ­  ­  ­  ­  ­  ­  ­  ­  ­  ­\",
    \"field\":\"YYYY_MM\",
    \"missingBucket\":false,
    \"missingBucketLabel\":\"Missing\",
    \"order\":\"desc\",
    \"orderBy\":\"1\",
    \"otherBucket\":false,
    \"otherBucketLabel\":\"Other\",
    \"row\":false,
    \"size\":12
  },
  \"schema\":\"split\",
  \"type\":\"terms\"
}

Which seems to be converted by spaces too by the browser but it actually looks like :

on my NotePad

and like :

on my output console

Would you have any idea of what is this ?

Regards
Guillaume