How to correctly use local JSON for Vega and Kibana?

Elastic Stack Kibana
1

HI, I'm experimenting a bit with the new Vega plugin for 6.2, but while it works pretty fine with external JSON files it fails when I try to achieve the same with a local stored JSON file.

I've tried with the treemap demo (https://vega.github.io/editor/#/examples/vega/treemap) and copy / pasting this in Kibana gives me the results I want when replacing the url to match the full path ("url": "https://vega.github.io/new-editor/data/flare.json")

However, when I save this file and store it on my local machine I get a 'no root' error message. The JSON itself is correct, and an exact copy of the file used when calling it with the above url.

I am running Kibana and ES on an Ubuntu server (16.04) and access rights to the folder where the JSON is stored is ok. I've tried by adding a root to the doc, but no difference. I also tried with different files, all the same problem.

My code is as below, the only difference with the working demo is with the url part, so I assume something is wrong there, but I can't find a working example using a locally stored JSON, so what is the correct syntax to use with local stored JSON?

     "data": [
    {
      "name": "tree",
      "url": "/data/kibana/flare.json",
      "transform": [
        {
          "type": "stratify",
          "key": "id",
          "parentKey": "parent"
        },
        {
          "type": "treemap",
          "field": "size",
          "sort": {"field": "value"},
          "round": true,
          "method": {"signal": "layout"},
          "ratio": {"signal": "aspectRatio"},
          "size": [{"signal": "width"}, {"signal": "height"}]
        }
      ]
    },
    {
      "name": "nodes",
      "source": "tree",
      "transform": [{ "type": "filter", "expr": "datum.children" }]
    },
    {
      "name": "leaves",
      "source": "tree",
      "transform": [{ "type": "filter", "expr": "!datum.children" }]
    }
  ]
2 Likes
2

Have you verified you can actually access the JSON file itself?

3

Yes, I can read the file, write to the file and validate it directly on the server. If I use the shell it opens as expected using the path mentioned. So availability of the file is not a problem as far as I can see.

4

What I mean is, is the file being served to where you can access it from an http call?

5

No, it's a local stored file, but on the same server as where Kibana and ES are running.
Does this mean you can only access JSON over http and not as local stored content?

Or does the JSON need to be stored in a folder relative to the Kibana ones?

Edit : I also tried with storing the files in my apache folder, and while I can then load the file correctly using http it still gives no root if I load it from my server instead of the vega server.

So if I use https://vega.github.io/new-editor/data/flare.json as url it works fine, using a copy of this json (bit to bit exact the same) will give me a 'no root' problem, while I can load it perfectly using http://my_server_ip/flare.json.

6

The url use are using, "/data/kibana/flare.json" means that Kibana itself has to serve this file over http(s). I'm not sure, but I don't think you can easily add arbitrary files to Kibana server in a reliable way. One option around this is to place your Kibana server behind an Nginx proxy (or similar), and have a collection of files served by that same proxy from a different URL path. For example, have your Kibana as https://example.org/kibana and your data files as https://example.org/data. Another way would be to host the files on a different domain, but in this case the domain must have properly configured CORS.

1 Like
7

yes, @nyuriks has a good point..

The url use are using, "/data/kibana/flare.json" means that Kibana itself has to serve this file over http(s).

Please try to do the following(Note you should have nodejs and npm):

  1. npm install http-server -g
    after that
  2. http-server path_to_your_folder_with_json_file --cors
    (only the folder no need to include the file name )
    after that try to access your json file in Vega
    "url": "http://localhost:8080/flare.json"

I was using it today..
Works like a charm :slight_smile:
Vega rules

1 Like
8

I see...

I have another application running on my 8080 port so maybe this one is conflicting with this logic.

Let me rephrase the question a bit then. Is it possible to load a json file not using http? Or basically are there other options than using a url (or ES overrule)?

something like "file": "/data/kibana/flare.json" instead of "url": "/data/kibana/flare.json" ?

9

Your browser wouldn't let you use anything other that http(s). If it did, someone could have accessed your local files without you knowing about it.

10

P.S. You can use -p 8081 to change the port - see https://www.npmjs.com/package/http-server

11

@Koen_Wouters Nope currently they support 3 types of data source:
Inline Data
Data from URL
Named Data Sources

If you wanna you can load your data inline (only if it is a small data set)

12

Some people are saying that they successfully managed to load a data using:
"url":"file:///path/data.json"
This is valid only for vega v2. You can test it

13

Thank you for addressing this - I had the same question - how can I get Kibana to have access to the JSON files required for some of the visualizations.

14

@bgiordano as mentioned before - you can use http(s) urls as data sources if you enable it in kibana.yaml file, but for many reasons including security, it will not allow you to use any other protocols like file://.

15

We have enabled - can access the vega.github.io with https just fine.
However, we need to have vega hosted locally within our own network - can't
go out to the internet - so far we can get the /schema just fine but we are
having issues with the /explorer/data access.

[image: www.novetta.com]

Barbara Giordano

Sr Data Computer Scientist

500 E Zack Street, 7th Floor

Tampa, FL 33602

Email bgiordano@novetta.com

Office 813-304-2942

Mobile 301-788-3946

16

@bgiordano please elaborate - what do you mean by /schema and /explorer/data? Are those paths to some well known data sources?

17

Yuri,

I apologize for not being clearer.

We need to reach Vega and Vega-lite from within our network, so we have
loaded it to a VM and are serving it. We can operate in the UW IDL editor
just fine.
We are running ES 5.5.0 or 5.5.1 currently.
We have installed the Vega Plug-in to Kibana and have edited the YML to
permit access via http/https.

Because we are using ES XPack security, we are finding that we need to
reach any external services from Kibana with HTTPS as well.

I have copied an example vis from the Vega gallery in to Kibana.
{
"$schema": "https://vega.github.io/schema/vega-lite/v2.1.json",
"width": 500,
"height": 300,
"data": {
"url": "data/us-10m.json",
"format": {
"type": "topojson",
"feature": "counties"
}
},
"transform": [{
"lookup": "id",
"from": {
"data": {
"url": "data/unemployment.tsv"
},
"key": "id",
"fields": ["rate"]
}
}],
"projection": {
"type": "albersUsa"
},
"mark": "geoshape",
"encoding": {
"color": {
"field": "rate",
"type": "quantitative"
}
}
}

When I change "$schema": "https://vega.github.io/schema/vega-lite/v2.1.json"

to point to my VM (ex. "https://my_vm_name/schema/vega-lite/v2.json")
that appears to work just fine.

However, when I change "url": "data/unemployment.tsv"
to "https://my_vm_name/editor/data/unemployment.tsv", I get 'loading
failed' error in kibana. Looking at the Network traffic, it reports a 200
for the file (success).

I can browse to https://my_vm_name/editor/data/unemployment.tsv" and see
the data.

We are still trying to figure out where things are falling apart.

Any ideas?

Thanks,
Barbara

[image: www.novetta.com]

Barbara Giordano

Sr Data Computer Scientist

500 E Zack Street, 7th Floor

Tampa, FL 33602

Email bgiordano@novetta.com

Office 813-304-2942

Mobile 301-788-3946

18

@bgiordano the schema string is an ID, it never needs to change, and Vega doesn't download it (or at least it shouldn't). The only reason it continues to work is because the schema id parsing is not very strict, but that may change.

The failure to load is very strange - your browser should be able to download https://my_vm_name/editor/data/unemployment.tsv directly, and use it just like it uses it from the Vega site. Make sure your CORS are properly set up - your browser uses XHR to access it, so if the server does not handle CORS correctly, it would fail.

19

We had been working the CORS setup - after a couple of tries found the
right solution. So things are working well now.
We enjoyed your webinar - you went very quickly so we will want to watch it
again and looking forward to your sankey blog.
Hope to meet you at Elasticon next week.

Barbara

[image: www.novetta.com]

Barbara Giordano

Sr Data Computer Scientist

500 E Zack Street, 7th Floor

Tampa, FL 33602

Email bgiordano@novetta.com

Office 813-304-2942

Mobile 301-788-3946

20

Yep, I will be there. The Sankey blog post is almost ready - I have it pre-published at https://nyurik.github.io/Vega-Sankey-Graph-for-Kibana/ but it should go onto Elastic blog pretty soon (I hope). The video and links should be available today or tomorrow as well. Thx!

1 Like