Unable to search documents in Kibana that is indexed using attachment plugin

[YuWatanabe]

I have indexed a base64 encoded attachment using latest ingest-attachment plugin.

My elsaticsearch and kibana environment is below.

[root@filebeat elasticsearch]# rpm -qa | grep kibana
kibana-5.0.1-1.x86_64
[root@filebeat elasticsearch]# rpm -qa | grep elasticsearch
elasticsearch-5.0.1-1.noarch

However, indexed document is not searchable from Kibana Discovery.

Doument definitely exists when I search from the query.

{
  "took": 6,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 2,
    "max_score": 1,
    "hits": [
      {
        "_index": "test",
        "_type": "test",
        "_id": "1",
        "_score": 1,
        "_source": {
          "date": "2016-11-17T18:50:00",
          "data": "H4sICFVwLVgAA2RhdGUudHh0AAvJKFXwyy9TMDRXMLCwMjWxMjFVCA1xVjAyMDTjAgBpNz/EHQAAAA==",
          "attachment": {
            "content_type": "application/gzip",
            "content_length": 0
          }
        }
      },
      {
        "_index": "test",
        "_type": "test",
        "_id": "3",
        "_score": 1,
        "_source": {
          "date": "2016-11-17T18:55:00",
          "data": "H4sICFVwLVgAA2RhdGUudHh0AAvJKFXwyy9TMDRXMLCwMjWxMjFVCA1xVjAyMDTjAgBpNz/EHQAAAA==",
          "attachment": {
            "content_type": "application/gzip",
            "content_length": 0
          },
          "subject": "My First attachment",
          "from": "creationline@localhost.localdomain",
          "to": "root@localhost.localdomain"
        }
      }
    ]
  }
}

Below is the query.

GET test/test/_search
{
  "query" : {
    "range" : {
      "date" : {
        "gte" : "now-1y"
      }
    }
  }
}

May I ask how can I debug the kibana query?

[tsullivan (Tim Sullivan)]

I was able to reproduce after indexing documents based on your post, and I found that I had to use an absolute time range in the date picker, because the dates are in the future in my timezone. It looks like if you use a relative date from some time in the past up until "now", then it doesn't show documents for the future.

Can you try using an absolute time range in the Kibana date picker? It looks like the dates in your documents might be in the future, and that could be throwing off Discover. You can also try Quick / Today

[YuWatanabe]

@tsullivan

Thanks for the support.

You are right. This was a timezone issue. I was able to work it out.

What was behind the scene was kibana automatically recognizes the client side timezone and adjust it when searches documents in discovery . So I use kibana in JST region and kibana adds +0900 to the timestamp field.

I just indexed a document this morning as below. Time is 2016-11-18 11:03:00

Discover shows timestamp in +0900 manner.

I didn't realize this issue till now because I usually use filebeat + logstash which automatically converts the time into UTC and then index the document.