Show most recent documents by entity

Elastic Stack Kibana
1

Hi,
I'm trying to use one of the features introduced in Kibana 7.3 for Maps.
I'm tracking vehicles, and because of that I have a lot of documents with Geo-points and indices of object.
When I'm adding a new layer I want to show most recent documents by entity, which in my case is object Id. I would like to show the most recent position of a vehicle.

But when I select this option I get an error saying: 2 of 3 shards failed.

When I looked at elastic search log I can see error saying "No mapping for @timestamp to sort on".

Can you help me somehow?

2

Hey @dominikw, if you go to Management -> Index Patterns and select the index pattern which contains your documents, do you see a label similar to the following at the top of the page?

41%20AM
Screen Shot 2019-08-20 at 10.16.41 AM.png2184×220 17.3 KB

3

Yes, this is exactly what I can see. That's why I'm totally confused by the log.

By the way, I tried to POST directly into elasticsearch service to obtain similar effect in response and it works.. I'm able to sort on the @timestamp there.

Is it possible that there is something wrong with Kibana?

4

@dominikw would you mind running the following queries in Dev Tools, replacing the foo index with the index you're pulling your documents from in the maps application and replying with the response?

GET /foo/_mapping
GET /foo/_field_caps
5

Sure thing. Mappings first.

{
  "vehicle_plugin" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "@version" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "data" : {
          "type" : "nested",
          "properties" : {
            "location" : {
              "type" : "geo_point"
            },
            "speed" : {
              "type" : "float"
            }
          }
        },
        "groups" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "handlerId" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "mversion" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "oid" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "stage" : {
          "type" : "long"
        },
        "timestamp" : {
          "properties" : {
            "eventTime" : {
              "type" : "long"
            },
            "ingestTime" : {
              "type" : "long"
            },
            "lastProcessingTime" : {
              "type" : "long"
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }
    }
  }
}

Field Caps causes error..

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "specified fields can't be null or empty"
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "specified fields can't be null or empty"
  },
  "status": 400
}
6

I have a lot of documents with Geo-points and indices of object.

Do you have multiple Elasticsearch indices? Which Elasticsearch indices does your Kibana index pattern match? Does each Elsaticsearch index map @timestamp as date.

I have seen shard failures like this where a Kibana index pattern matches multiple Elastic search indices and the mapping type is not consistent between the Elasticsearch indices. This results in shard failures as one index can not successfully complete the query because the mappings are different and the difference causes an error

7

Field Caps causes error

This is because the field capabilities request did not specify the required argument fields.

Try running
GET /foo/_field_caps?fields=@timestamp or GET /foo/_field_caps?fields=*

8

Unfortunately no, I have only one index matching my Kibana index pattern.

9

Thanks for suggestion with field caps. Here is the output.

{
  "indices" : [
    "vehicle_plugin"
  ],
  "fields" : {
    "data" : {
      "nested" : {
        "type" : "nested",
        "searchable" : false,
        "aggregatable" : false
      }
    },
    "_index" : {
      "_index" : {
        "type" : "_index",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_feature" : {
      "_feature" : {
        "type" : "_feature",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "timestamp.ingestTime" : {
      "long" : {
        "type" : "long",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "oid" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "type" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "timestamp.lastProcessingTime" : {
      "long" : {
        "type" : "long",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "timestamp.eventTime" : {
      "long" : {
        "type" : "long",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_ignored" : {
      "_ignored" : {
        "type" : "_ignored",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "type.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "handlerId" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "@version" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "mversion.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_version" : {
      "_version" : {
        "type" : "_version",
        "searchable" : false,
        "aggregatable" : false
      }
    },
    "mversion" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "timestamp" : {
      "object" : {
        "type" : "object",
        "searchable" : false,
        "aggregatable" : false
      }
    },
    "_routing" : {
      "_routing" : {
        "type" : "_routing",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "groups.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_type" : {
      "_type" : {
        "type" : "_type",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "groups" : {
      "text" : {
        "type" : "text",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "data.speed" : {
      "float" : {
        "type" : "float",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "@version.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "oid.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_seq_no" : {
      "_seq_no" : {
        "type" : "_seq_no",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "@timestamp" : {
      "date" : {
        "type" : "date",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "stage" : {
      "long" : {
        "type" : "long",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_field_names" : {
      "_field_names" : {
        "type" : "_field_names",
        "searchable" : true,
        "aggregatable" : false
      }
    },
    "data.location" : {
      "geo_point" : {
        "type" : "geo_point",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "_source" : {
      "_source" : {
        "type" : "_source",
        "searchable" : false,
        "aggregatable" : false
      }
    },
    "_id" : {
      "_id" : {
        "type" : "_id",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "handlerId.keyword" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    }
  }
}
10

Just noticed that data is nested and kibana does not support nested types. I wonder if this is causing some problems.

What does the Elasticsearch request/response look like when you enable most recent documents by entity? Follow the instructions here to view the request and response.

11

Oh, I was not aware about "Inspect" feature, nice to know this.

Seems like you were right at the beginning, Nathan. Even though Index Pattern Management was not showing more indices matching the pattern, I had some leftovers in elastic search which were satisfying the pattern. So the map viewer itself is using the regex pattern, not the actual index name for the requests.
Thanks for help, everything seems fine now! I really appreciate your effort

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.