[MAPS][scripted field]Heading and lat/lon not in same doc, any way to merge them?

[Sjaak01 (Sjaak)]

Hi,

I am collecting lat/lon data for a moving object containing three fields.

{
"object": "name",
"geo": "lat lon",
"@timestamp": 12:00:00
}

I am also receiving heading information from a different source.

{
"object": "name",
"heading": "200",
"@timestamp": 11:23:00
}

The lat/lon is collected hourly, heading at random intervals.

I want to use the heading data to plot the direction of an icon on the MAPS app. But because the data is not in the same document this doesn't look like a straight forward job. I tried using term joins but I don't think they can do what I want.

Is there any way Kibana can somehow pull the closest heading data (based on @timestamp) into the document containing the lat/lon?

[Aaron_Caldwell]

Hello Sjaak,

If you're using Logstash to ingest your data, the filter plugin may work for you. You would just pull the heading data from one index into the new docs being inserted into their index.

In the near future, this should be an option using Elasticsearch Ingest Enrichment which is in-progress now and described here. Effectively this would allow you to ingest data and "decorate" the data on ingest using data from a different index. Feel free to keep an eye on this issue!

Regards,
Aaron

[Sjaak01 (Sjaak)]

That is a very good idea. I spent today making a demo and while it could work, the queries supported by that filter are very limited. As far as I can tell bool queries are not supported (not mentioned anywhere and Logstash reports no errors, just doesn't work) and neither is KQL.

This means there is no way to search for documents older than the Logstash document. My data is not real time and I cannot be sure Logstash will ingest everything oldest first so without some kind of range filter it might retrieve the wrong data.