Vega is not showing all the data (very sparse)

Hi All

Does anybody know how to control level of detail in Vega map projections?

I have created a Vega visualization for putting lines between sources and destinations (rule mark type) on a map (definition below), something like in this example, but for the whole world:

But unlike the example, in my case it doesn't show all the pairs returned by the query, only some part of it. When I create data table aggregation with source and destinations (as geohashes or corresponding city names), there're more pairs. Also, when I increase time range, some old pairs disappear from the map, so it's definitely not data issue. I could understand that if it was about plotting 1000s of pairs, but I have only tens of them.
So the question is, how to control the level of detail in Vega or make it plotting more points/pairs/lines???

My Vega visualization (3 layers: source-destination lines and circles for sources and destinations):

{
  "config": {
    "kibana": {
      "type": "map",
      "latitude": 40.7,
      "longitude": -74,
      "zoom": 0,
      "minZoom": 0,
      "maxZoom": 18,
      "delayRepaint": true
    }
  },
  "$schema": "https://vega.github.io/schema/vega-lite/v2.json",
  "title": "Comm map",
  "layer": [
    {
      "data": {
        "url": {
          "index": "some_index*",
          "body": {
            "query": {
              "bool": {
                "must": [
                  {
                    "query_string": {
                      "query": "_exists_:src_GeoIP AND _exists_:dest_GeoIP"
                    }
                  },
                  {"range": {"timestamp": {"%timefilter%": true}}}
                ]
              }
            }
          }
        },
        "format": {"property": "hits.hits"}
      },
      "projection": {"type": "mercator"},
      "mark": {
        "type": "rule",
        "color": "#ff0000",
        "strokeWidth": 9,
        "opacity": 0.1
      },
      "encoding": {
        "longitude": {
          "field": "_source.src_GeoIP.location.lon",
          "type": "quantitative",
          "scale": {"nice": true}
        },
        "latitude": {
          "field": "_source.src_GeoIP.location.lat",
          "type": "quantitative",
          "scale": {"nice": true}
        },
        "longitude2": {
          "field": "_source.dest_GeoIP.location.lon",
          "type": "quantitative",
          "scale": {"nice": true}
        },
        "latitude2": {
          "field": "_source.dest_GeoIP.location.lat",
          "type": "quantitative",
          "scale": {"nice": true}
        }
      }
    },
    {
      "data": {
        "url": {
          "index": "some_index*",
          "body": {
            "query": {
              "bool": {
                "must": [
                  {
                    "query_string": {
                      "query": "_exists_:src_GeoIP AND _exists_:dest_GeoIP"
                    }
                  },
                  {"range": {"timestamp": {"%timefilter%": true}}}
                ]
              }
            }
          }
        },
        "format": {"property": "hits.hits"}
      },
      "projection": {"type": "mercator"},
      "mark": "circle",
      "encoding": {
        "longitude": {
          "field": "_source.src_GeoIP.location.lon",
          "type": "quantitative"
        },
        "latitude": {
          "field": "_source.src_GeoIP.location.lat",
          "type": "quantitative"
        },
        "size": {"value": 100},
        "color": {"value": "blue"}
      }
    },
    {
      "data": {
        "url": {
          "index": "some_index*",
          "body": {
            "query": {
              "bool": {
                "must": [
                  {
                    "query_string": {
                      "query": "_exists_:src_GeoIP AND _exists_:dest_GeoIP"
                    }
                  },
                  {"range": {"timestamp": {"%timefilter%": true}}}
                ]
              }
            }
          }
        },
        "format": {"property": "hits.hits"}
      },
      "projection": {"type": "mercator"},
      "mark": "circle",
      "encoding": {
        "longitude": {
          "field": "_source.dest_GeoIP.location.lon",
          "type": "quantitative"
        },
        "latitude": {
          "field": "_source.dest_GeoIP.location.lat",
          "type": "quantitative"
        },
        "size": {"value": 100},
        "color": {"value": "red"}
      }
    }
  ]
}

@nyuriks can you please help here ?

Thanks
Rashmi

@vsmirnov hi, I really need to see the data that you are trying to plot. Without it it would be very hard to debug the issue. Could you use JSON.stringify(VEGA_DEBUG.vega_spec) command from the browser debug tool (console) to copy your graph including the data, and attach it as a zip file to the issue? Make sure to expand the output to include everything. Thanks! cc: @rashmi

Hi Yuri

I'm terribly sorry, I forgot to include "size" parameter in Elasticsearch query, so it was showing only 10 results :smiley:
Now works perfectly!

Thank you for your response!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.