Documents are stored on elasticsearch, can be checked via query, but Kibana does not find it in the discovery tab

[toshi78]

Hi!

I'm facing an issue on Kibana: Looks like statistics show that a number of documents are available:

However, on the Discover tab, it does not find any document, no matter the date range set:

No results match your search criteria

But when I perform the following query (inside the Dev Tools tab), it find my documents:

GET myMapping/_search
{
  "query": {
    "match_all": {}
  }
}

Result:

{
  "took" : 11,
  "timed_out" : false,
  "_shards" : {
    "total" : 9,
    "successful" : 9,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 150,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      { ...

My mapping is accepted by elasticsearch, and when I setup Kibana's Index, my index and Date field are well recognized:

curl -u elastic -XPUT "https://localhost:9200/myMapping" -d @cfg/elastic/myMapping.json -H 'Content-Type: application/json' --cacert docker/ca.crt
Enter host password for user 'elastic':

{"acknowledged":true,"shards_acknowledged":true,"index":"myMapping"}

What's the problem here? How can I find what's wrong?

Thank you in advance.

Note: Im using Docker for the ELK infrastructure, and has 3 nodes. The mapping's settings is set to:

"settings": {
    "index" : {
        "number_of_shards" : 9,
        "number_of_replicas" : 3
    }
}

[wylie (Wylie Conlon)]

It looks like you have set the field collectedTime as the primary time field for your Kibana index pattern: almost all Kibana apps use the time field as a filter on the set of visible documents. It sounds like this might not be what you were expecting, so here are the options I can think of for you:

• Make sure your time fields are being indexed correctly in Elasticsearch
• If you don't want to filter any documents by time, you can choose not to have a primary time field. You would need to delete the index pattern and re-create it without a time field.
• You can change the primary time field by deleting and re-creating the index pattern

[toshi78]

I've re-created the index pattern without a time field, and I do see now my documents.

However, I need to use the collectedTime field as a primary time field: How can I check what's wrong with it?

Here's a part of my mapping about the Date fields:

"mappings": {
    "properties": {
	"collectedTime": {
	    "type": "date",
	    "format": "yyyy-MM-dd HH:mm:ss.SSSSSSZ"
	},
	"publish_date": {
	    "type": "date",
	    "format": "yyyy-MM-dd HH:mm:ss.SSSSSSZ"
	},
...

And an example of a stored document:

{
  "_index": "myMapping",
  "_type": "_doc",
  "_id": "wcGacHIBy-QMce-LlAim",
  "_version": 1,
  "_score": 0,
  "_source": {
    "collectedTime": "2020-06-01 17:57:14.792395+0000",
    "publishDate": "2020-05-28 07:01:00.000000+0100",
    ...

Note: Kibana well recognized the date format when I setup the Time filter field:

[toshi78]

Well, it suddently worked after several re-creation of the index... I just had a warning when I opened the Discovery tab:

"fac45350-a42f-11ea-b147-dd0899ebf563" is not a configured index pattern ID
Showing the default index pattern: "myMapping" (46d9eb10-a435-11ea-b147-dd0899ebf563)

Any idea of what happened? Maybe the Discovery tab was "stucked" with an old index pattern?

[wylie (Wylie Conlon)]

To use a field as a time field, every document needs to have correct timestamps. I would recommend checking it by using a time range query:

POST myMapping/_search
{
  "query": {
    "range": {
      "collectedTime": {
        "gte": "now-7d",
        "lte": "now"
      }
    }
  },
  "size": 10
}

I think you can work around that error message by switching to a different pattern and back, or by clearing the URL parameters.