Visualizing fading effect of geo points in kibana

Elastic Stack Kibana
1

I am trying to display data in kibana which has geo location attribute in it. Thus i am trying to display it using Tile Map.

I am able to see the points on the map but what i want is to show in a fading effect. That is the most recent data should have higher value for heat map effect on the map while the oldest entry the least value for heat map color.

Is there a way i can do this?

The data that i import using logstash is

8702.04,11.3045,46.1689,255.17,345.4,4.23,400fe2,EZY87VZ,false,1.49519499E9,2017-05-19 13:56:30,United Kingdom
9982.2,11.1508,46.5768,260.3,345.11,8.13,400fe2,EZY87VZ,false,1.49519517E9,2017-05-19 13:59:29,United Kingdom
10972.8,10.9876,46.9772,250.97,337.59,0.33,400fe2,EZY87VZ,false,1.49519535E9,2017-05-19 14:02:30,United Kingdom
11582.4,10.7482,47.3419,245.31,334.67,-0.33,400fe2,EZY87VZ,false,1.495195529E9,2017-05-19 14:05:29,United Kingdom
11590.02,10.3498,47.6354,251.07,316.33,-0.33,400fe2,EZY87VZ,false,1.495195709E9,2017-05-19 14:08:29,United Kingdom

As you can see, this dataset has a date attribute as the second last field, i want to use this date to show somehow the last record with date "2017-05-19 14:08:29" as the darkest spot(red) and other lighter and gradually getting lighter in color and thus the fading effect.

Right now all the points are visible on map but with same intensity of color as shown in below snapshot

1078×695 636 KB

The metrics used for creating this is

376×620 16.7 KB

Can someone suggest how can i achieve this.
Thanks

2

This is not currently a feature of our visualizations. Feel free to open up a feature request here: https://github.com/elastic/kibana/issues

3

Try doing a max aggregation on the timestamp field, it should do something like what you want.

4

By adding a new field (myscore) in the dataset and using that i could somehow achieve the fading effect.

Dataset

8702.04,11.3045,46.1689,255.17,345.4,4.23,400fe2,EZY87VZ,false,1.49519499E9,2017-05-19 13:56:30,United Kingdom,6
9982.2,11.1508,46.5768,260.3,345.11,8.13,400fe2,EZY87VZ,false,1.49519517E9,2017-05-19 13:59:45,United Kingdom,22
10972.8,10.9876,46.9772,250.97,337.59,0.33,400fe2,EZY87VZ,false,1.49519535E9,2017-05-19 14:03:30,United Kingdom,51
11582.4,10.7482,47.3419,245.31,334.67,-0.33,400fe2,EZY87VZ,false,1.495195529E9,2017-05-19 14:07:29,United Kingdom,111
11590.02,10.3498,47.6354,251.07,316.33,-0.33,400fe2,EZY87VZ,false,1.495195709E9,2017-05-19 14:14:15,United Kingdom,842

So as you can see i have added another field (myscore) in the last.

Capture.jpg1294×928 846 KB

The metric used for creating this is

Capture.PNG372×886 23.8 KB

Options

5

@warkolm @tsmalley

The above solution that i have shared was a work around that i could do.

But this is what i am not looking for.

I want to use the original dataset, which has a timestamp field in it. I then want to utilize that timestamp to reflect the @timestamp in elasticsearch....which can be done.
Then i want to fetch records using a query and custom score function such that the most recent ones have highest score and the score gradually decreases with timestamp. This again can be done.

Refer

Using this discussion i created a score function for timestamp.

So now i have the original dataset without the myscore field, and using my custom query i get results which has score updated according to timestamp.

Query

    curl -XGET 'localhost:9200/test/_search?pretty' -H 'Content-Type: application/json' -d'
    {
        "query": {
            "function_score": {
                "gauss": {
                    "@timestamp": {
                          "origin": "1496061764181", 
                          "scale": "1ms", 
                          "decay" : 0.5 
                    }
                }
            }
        }
    }
    '

But the problem is that i cannot achieve this fading effect when i consider _score field. Refer below screenshots.

1468×892 743 KB

Options

Why is _score field on hovering over the point not reflecting the score? And why can't the fading effect be achieved with this?

Please can you suggest what can be done.

6

@thomasneirynck - mind providing some insights?

7

@Rahul_Lao

Can you attach the request/response of that last query?

Open the spy-panel using the little grey arrow on the bottom left. It has two tabs; one with the ES request, and one with the ES response.

thx,

8

Default – no query in kibana query bar

Elasticsearch request body

{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*",
            "analyze_wildcard": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1494767233584,
              "lte": 1497359233585,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "geohash_grid": {
        "field": "location",
        "precision": 4
      },
      "aggs": {
        "1": {
          "top_hits": {
            "_source": "_score",
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        },
        "3": {
          "geo_centroid": {
            "field": "location"
          }
        }
      }
    }
  }
}

Elasticsearch response body

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pp",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764169
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.16889998316765,
              "lon": 11.304499972611666
            }
          },
          "key": "u20n",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0ps",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764180
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 47.341899974271655,
              "lon": 10.748199932277203
            }
          },
          "key": "u0rt",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pt",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764181
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 47.63539994135499,
              "lon": 10.349799860268831
            }
          },
          "key": "u0rq",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pr",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764178
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.97719993069768,
              "lon": 10.98759999498725
            }
          },
          "key": "u0rg",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pq",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764171
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.57679993659258,
              "lon": 11.150799840688705
            }
          },
          "key": "u0rb",
          "doc_count": 1
        }
      ]
    }
  },
  "status": 200
}
9

With Query in kibana query bar

Elasticsearch request body

{
  "query": {
    "bool": {
      "must": [
        {
          "function_score": {
            "gauss": {
              "@timestamp": {
                "origin": "1496061764181",
                "scale": "1ms",
                "decay": 0.5
              }
            }
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1494767539343,
              "lte": 1497359539343,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "size": 0,
  "_source": {
    "excludes": []
  },
  "aggs": {
    "2": {
      "geohash_grid": {
        "field": "location",
        "precision": 4
      },
      "aggs": {
        "1": {
          "top_hits": {
            "_source": "_score",
            "size": 1,
            "sort": [
              {
                "@timestamp": {
                  "order": "desc"
                }
              }
            ]
          }
        },
        "3": {
          "geo_centroid": {
            "field": "location"
          }
        }
      }
    }
  }
}

Elasticsearch response body

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pp",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764169
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.16889998316765,
              "lon": 11.304499972611666
            }
          },
          "key": "u20n",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0ps",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764180
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 47.341899974271655,
              "lon": 10.748199932277203
            }
          },
          "key": "u0rt",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pt",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764181
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 47.63539994135499,
              "lon": 10.349799860268831
            }
          },
          "key": "u0rq",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pr",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764178
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.97719993069768,
              "lon": 10.98759999498725
            }
          },
          "key": "u0rg",
          "doc_count": 1
        },
        {
          "1": {
            "hits": {
              "total": 1,
              "max_score": null,
              "hits": [
                {
                  "_index": "test",
                  "_type": "logs",
                  "_id": "AVxUOuMaCj46ZjiRI0pq",
                  "_score": null,
                  "_source": {},
                  "sort": [
                    1496061764171
                  ]
                }
              ]
            }
          },
          "3": {
            "location": {
              "lat": 46.57679993659258,
              "lon": 11.150799840688705
            }
          },
          "key": "u0rb",
          "doc_count": 1
        }
      ]
    }
  },
  "status": 200
}

Screenshot - _score is empty in both cases

10

Even running query in discover tab doesnot update the score

image.png1920×1200 370 KB

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.