Hi,
Not sure if its completely a Kibana issue but for some reason its not showing any data using the "discover" part.
In the "index patterns" the index has been setup with a specific date field to be used for any time filters.
Data is being indexed fine and when I do a manual search the data shows up fine.
The indexes are time based so setup the pattern with a * at the end.
Is there a way I can debug this further? I'm bit at a loss.
Running ES + Kibana 7.3.0
Hi @electrical1!
Welcome to the discussion boards.
Can you provide the mappings for your index? Thanks!
Hey @electrical1,
The best way to debug this further is to use the "Inspect" feature. From the Discover app, click the "Inspect" link after attempting a search which returns 0 results.
From there, a flyout will appear on the righthand side of the screen. Click the "Request" tab there to see the request that Kibana issued to Elasticsearch:
You can also view the response too, but I'm guessing there won't be anything useful there.
Here is the mapping:
{
"mapping": {
"properties": {
"bandwidthPlan": {
"type": "keyword"
},
"bandwidthPlanId": {
"type": "short"
},
"bandwidthPlanPartnerId": {
"type": "short"
},
"beamId": {
"type": "short"
},
"beamName": {
"type": "keyword"
},
"beamPool": {
"type": "keyword"
},
"beamPoolId": {
"type": "short"
},
"beamPool_downloadRate": {
"type": "integer"
},
"beamPool_uploadRate": {
"type": "integer"
},
"borrowing": {
"type": "keyword"
},
"carrierId": {
"type": "short"
},
"creationTime": {
"type": "date",
"format": "YYYY-MM-DD'T'HH:mm:ss'Z'"
},
"diagnostics": {
"type": "boolean"
},
"endpointPolicy": {
"type": "keyword"
},
"globalPool": {
"type": "keyword"
},
"globalPoolId": {
"type": "short"
},
"globalPool_downloadRate": {
"type": "integer"
},
"globalPool_uploadRate": {
"type": "integer"
},
"hub": {
"type": "keyword"
},
"hubId": {
"type": "short"
},
"hubPool": {
"type": "keyword"
},
"hubPoolId": {
"type": "short"
},
"hubPool_downloadRate": {
"type": "integer"
},
"hubPool_uploadRate": {
"type": "integer"
},
"hub_downloadRate": {
"type": "integer"
},
"hub_uploadRate": {
"type": "integer"
},
"isNonBillable": {
"type": "boolean"
},
"oid": {
"type": "long"
},
"parentOid": {
"type": "long"
},
"partnerId": {
"type": "short"
},
"partnerType": {
"type": "keyword"
},
"region": {
"type": "keyword"
},
"satellite": {
"type": "keyword"
},
"satelliteId": {
"type": "short"
},
"sessionId": {
"type": "keyword"
},
"slaId": {
"type": "short"
},
"slaLegacyId": {
"type": "short"
},
"slaLegacy_downloadRate": {
"type": "integer"
},
"slaLegacy_fairFactor": {
"type": "short"
},
"slaLegacy_uploadRate": {
"type": "integer"
},
"sla_downloadRate": {
"type": "integer"
},
"sla_uploadRate": {
"type": "integer"
},
"state": {
"type": "keyword"
},
"subPool": {
"type": "keyword"
},
"subPoolId": {
"type": "short"
},
"subPoolName": {
"type": "keyword"
},
"subPool_downloadRate": {
"type": "integer"
},
"subPool_uploadRate": {
"type": "integer"
},
"subpoolClass": {
"type": "keyword"
},
"subscriberId": {
"type": "keyword"
},
"svnoInterconnect": {
"type": "boolean"
},
"updateTime": {
"type": "date",
"format": "YYYY-MM-DD'T'HH:mm:ss.SSS'Z'"
},
"usagePolicyId": {
"type": "short"
},
"vlan": {
"type": "long"
},
"voip": {
"type": "boolean"
}
}
}
}@Larry_Gregory I've tried that yeah but didn't find anything obvious wrong with it.
Happy to paste the content of the inspect request part if needed.
The only other thought I have is related to timezone. If you expand your search range in both directions (e.g., 2 days ago -> 2 days from now), do you see any data? It's possible that Kibana is translating your query into a timezone that you aren't expecting it to use.
I did yeah. I searched in the last 30 days and also in "this month". no luck.
It can't hurt, but I suspect it won't be terribly helpful without access to the underlying indices too, which you won't be able to provide 
@Larry_Gregory @alisongoryachev any idea's? Bit at a loss here.
Sorry, no new ideas -- are you still able to provide the contents of the inspect request part? We'll take a look at that and see if anything jumps out.
As a quick sanity check, can you run the query from that inspect section manually and see if it returns anything? I know you mentioned you can query the indices manually, but I'm interested to see how your manual query differs from the one that Kibana tries to run.
Here is the request part.
The manual request is without any filtering or sorts which does give back all the data without any issues.
{
"version": true,
"size": 500,
"sort": [
{
"updateTime": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes": []
},
"aggs": {
"2": {
"date_histogram": {
"field": "updateTime",
"fixed_interval": "12h",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
"*"
],
"script_fields": {},
"docvalue_fields": [
{
"field": "creationTime",
"format": "date_time"
},
{
"field": "updateTime",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"range": {
"updateTime": {
"format": "strict_date_optional_time",
"gte": "2019-07-28T17:02:10.285Z",
"lte": "2019-08-27T17:02:10.285Z"
}
}
}
],
"filter": [
{
"match_all": {}
}
],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
}
}
Response:
{
"took": 36,
"timed_out": false,
"_shards": {
"total": 264,
"successful": 264,
"skipped": 263,
"failed": 0
},
"hits": {
"total": 0,
"max_score": null,
"hits": []
},
"aggregations": {
"2": {
"buckets": []
}
},
"status": 200
}Thanks for the response. Ok, I have a theory. Taking another look at the mapping you provided earlier, I noticed the following:
"creationTime": {
"type": "date",
"format": "YYYY-MM-DD'T'HH:mm:ss'Z'"
},
When Kibana builds the query to Elasticsearch, it provides the following range query:
"query": {
"bool": {
"must": [
{
"range": {
"updateTime": {
"format": "strict_date_optional_time",
"gte": "2019-07-28T17:02:10.285Z",
"lte": "2019-08-27T17:02:10.285Z"
}
}
}
]
}
},
Kibana is hard-coded to expect a date field that conforms to the built-in strict_date_optional_time format, which your field does not conform to.
If you change the format of that field to be something like yyyy-MM-dd'T'HH:mm:ssZZ (which is the built-in strict_date_time_no_millis format), does the discover app populate the way you'd expect it to?
Just to clarify, you need me to change the format of how we store the data?
Any specific reason why kibana is hard coded like that? doesn't make it very flexible to use.
Did some more testing today.
When i do a manual query using Postman using the 2019-08-27T17:02:10.285Z format it doesn't work with strict_date_optional_time but does work if i use the format YYYY-MM-dd'T'HH:mm:ss.SSS'Z'
Next issue I found now is that the aggregation does something very weird.
The month, day, hour and minute values are correct but all my year values are 1970.
For example:
"aggregations": {
"usage": {
"buckets": [
{
"key_as_string": "1970-03-13T00:00:00.000Z",
"key": 6134400000,
"doc_count": 32
},
{
"key_as_string": "1970-03-21T00:00:00.000Z",
"key": 6825600000,
"doc_count": 2
},
{
"key_as_string": "1970-03-29T00:00:00.000Z",
"key": 7516800000,
"doc_count": 4
},
{
"key_as_string": "1970-04-10T00:00:00.000Z",
"key": 8553600000,
"doc_count": 2
},
{
"key_as_string": "1970-04-23T00:00:00.000Z",
"key": 9676800000,
"doc_count": 2
},
{
"key_as_string": "1970-04-25T00:00:00.000Z",
"key": 9849600000,
"doc_count": 2
},
{
"key_as_string": "1970-05-07T00:00:00.000Z",
"key": 10886400000,
"doc_count": 80
},
{
"key_as_string": "1970-05-09T00:00:00.000Z",
"key": 11059200000,
"doc_count": 2
},
{
"key_as_string": "1970-05-13T00:00:00.000Z",
"key": 11404800000,
"doc_count": 2
},
{
"key_as_string": "1970-05-14T00:00:00.000Z",
"key": 11491200000,
"doc_count": 4
},
{
"key_as_string": "1970-05-16T00:00:00.000Z",
"key": 11664000000,
"doc_count": 8
},
{
"key_as_string": "1970-05-17T00:00:00.000Z",
"key": 11750400000,
"doc_count": 4
},
{
"key_as_string": "1970-05-24T00:00:00.000Z",
"key": 12355200000,
"doc_count": 4
},
{
"key_as_string": "1970-05-28T00:00:00.000Z",
"key": 12700800000,
"doc_count": 2
},
{
"key_as_string": "1970-06-06T00:00:00.000Z",
"key": 13478400000,
"doc_count": 2
},
{
"key_as_string": "1970-06-07T00:00:00.000Z",
"key": 13564800000,
"doc_count": 2
},
{
"key_as_string": "1970-06-13T00:00:00.000Z",
"key": 14083200000,
"doc_count": 6
},
{
"key_as_string": "1970-06-14T00:00:00.000Z",
"key": 14169600000,
"doc_count": 2
},
{
"key_as_string": "1970-06-15T00:00:00.000Z",
"key": 14256000000,
"doc_count": 2
},
{
"key_as_string": "1970-06-17T00:00:00.000Z",
"key": 14428800000,
"doc_count": 8
},
{
"key_as_string": "1970-06-18T00:00:00.000Z",
"key": 14515200000,
"doc_count": 188
},
{
"key_as_string": "1970-06-19T00:00:00.000Z",
"key": 14601600000,
"doc_count": 14
},
{
"key_as_string": "1970-06-20T00:00:00.000Z",
"key": 14688000000,
"doc_count": 8
},
{
"key_as_string": "1970-06-21T00:00:00.000Z",
"key": 14774400000,
"doc_count": 4
},
{
"key_as_string": "1970-06-24T00:00:00.000Z",
"key": 15033600000,
"doc_count": 2
},
{
"key_as_string": "1970-06-27T00:00:00.000Z",
"key": 15292800000,
"doc_count": 2
},
{
"key_as_string": "1970-06-28T00:00:00.000Z",
"key": 15379200000,
"doc_count": 8
},
{
"key_as_string": "1970-06-30T00:00:00.000Z",
"key": 15552000000,
"doc_count": 2
},
{
"key_as_string": "1970-07-01T00:00:00.000Z",
"key": 15638400000,
"doc_count": 2
},
{
"key_as_string": "1970-07-08T00:00:00.000Z",
"key": 16243200000,
"doc_count": 6
},
{
"key_as_string": "1970-07-09T00:00:00.000Z",
"key": 16329600000,
"doc_count": 2
}
]
}
}
Any idea what could be causing this?
So. finally found it. We had YYYY in our mappings for the year which of course should have been yyyy which has been causing issues.
Now its all working fine 
This issue can be closed.
Thank you so much for your support @Larry_Gregory @alisongoryachev
Ah, great find! Sorry I didn't catch that myself
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.