# Courier fetch error when creating index pattern (5.5.1) **URL:** \ **Category:** Kibana\ **Created:** [November 9, 2017, 12:00pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038 "2017-11-09T12:00:03Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![mg6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mg6/32/24018_2.png) [@mg6](https://discuss.elastic.co/u/mg6)\ **Post date:** [November 9, 2017, 12:00pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/1 "2017-11-09T12:00:03Z")
Hello, We have successfully deployed ELK 5.5.1 stack to our GCE cluster. Elasticsearch seems to run fine and is able to get the data through Logstash (indexes are created and populated correctly; cluster state is green). However, Kibana is not able to create the index pattern with default settings. After pressing the _Create_ button in _Configure an index pattern_ view, there is an XHR request to `_mget` endpoint which fails with `400 Bad Request` error, and the following fatal error message is shown: ```auto Courier Fetch Error: unhandled courier request error: [action_request_validation_exception] Validation Failed: 1: id is missing for doc 0; Version: 5.5.3 Build: 15460 Error: unhandled courier request error: [action_request_validation_exception] Validation Failed: 1: id is missing for doc 0; handleError@https://host/bundles/kibana.bundle.js?v=15460:229:17545 (...) ``` Other related information: - X-Pack plugin is disabled. - The same error occurs for Kibana versions 5.5.1 up to 5.5.3. - Kibana is hosted from a subpath by a reverse proxy. Requests are rewritten correctly as `/subpath/` → `/` in the logs. Please let me know if I should provide more details. Thanks! **XHR requests leading to exception** ```auto POST /es_admin/.kibana/index-pattern/logstash-*/_create {"title":"logstash-*","timeFieldName":"@timestamp","notExpandable":true} 200 OK ok ``` ```auto POST /es_admin/_mget {"docs":[{"_index":".kibana","_type":"index-pattern"}]} 400 Bad Request {"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is missing for doc 0;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: id is missing for doc 0;"},"status":400} ``` ---
**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\ **Post date:** [November 9, 2017, 11:03pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/2 "2017-11-09T23:03:42Z")
Hi Maciej, thanks for sharing such great details about your issue! This is very odd, because the error is basically saying that the `_id` property is expected and Kibana _should_ be sending that along when you create an index pattern. Could you share a screenshot of how you're configuring your index pattern in the "Configure an index pattern" view? Thanks, CJ ---
**Author:** ![mg6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mg6/32/24018_2.png) [@mg6](https://discuss.elastic.co/u/mg6)\ **Post date:** [November 9, 2017, 11:43pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/3 "2017-11-09T23:43:09Z")
Sure! Basically, the default parameters are used. ![27](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac7d62b6e940e57ccc045803deaeffe042c9fc57.png) I forgot to mention; we also tried creating the index pattern by hand with the following queries: ```auto POST .kibana/index-pattern/logstash-*/_create {"title":"logstash-*","timeFieldName":"@timestamp"} POST .kibana/index-pattern/logstash-*/_create {"title":"logstash-*","timeFieldName":"@timestamp","notExpandable":true} ``` However, both lead to a distorted user interface (while `_cat/health` reports 100% active shards): ![33](https://us1.discourse-cdn.com/elastic/original/3X/e/1/e1844ea1e25955108bd3c2e96727b46d139b6782.png) and the following error is present in the console: ```auto Error: indexPattern.fields.byName is undefined isSortable@https:// __host__ / __subpath__ /bundles/kibana.bundle.js?v=15460:250:4092 (...) ``` so we reverted this with: ```auto DELETE .kibana/index-pattern/logstash-* ``` ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [November 13, 2017, 8:29pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/4 "2017-11-13T20:29:09Z")
Hi @mg6, Hmm. I'm not quite sure what's going on, but can you get yourself back into the broken environment and show me the results of: ```auto POST .kibana/index-pattern/_search { } ``` Feel free to `***` out any sensitive data if necessary. Thanks ---
**Author:** ![mg6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mg6/32/24018_2.png) [@mg6](https://discuss.elastic.co/u/mg6)\ **Post date:** [November 14, 2017, 10:13am UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/5 "2017-11-14T10:13:30Z")
Hello @chrisronline, This is the output in distorted UI state: ```auto { "took": 2, "timed_out": false, "_shards": { "total": 1, "successful": 1, "failed": 0 }, "hits": { "total": 1, "max_score": 1, "hits": [ { "_index": ".kibana", "_type": "index-pattern", "_id": "logstash-*", "_score": 1, "_source": { "title": "logstash-*", "timeFieldName": "@timestamp", "notExpandable": true } } ] } } ``` Regards ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [November 14, 2017, 2:51pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/6 "2017-11-14T14:51:46Z")
Thanks @mg6 Below is an animated gif trying to reproduce this problem. I understand why the latter error occurs (`Error: indexPattern.fields.byName is undefined`) but I don't understand why Kibana isn't auto-fixing itself. In the gif below, you'll see that I'm attempting to follow the same steps as you, but after I load the discovery area, you'll see a series of network requests that fixes the index pattern, by populating it with the appropriate fields from the indices in ES. Can you compare what you see when you follow these steps to what is in the gif and let me know what's different? Thanks! ![](https://dl.dropboxusercontent.com/s/oh2urkf93nll69h/5.5.3_index_pattern_creation.gif?dl=0) ---
**Author:** ![mg6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mg6/32/24018_2.png) [@mg6](https://discuss.elastic.co/u/mg6)\ **Post date:** [November 14, 2017, 7:28pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/7 "2017-11-14T19:28:42Z")
Hello Chris, Thank you for your step-by-step demonstration – it enabled me to finally locate the root cause of the problem and fix the issue. As it turned out, Kibana was correctly trying to fix itself – the problem was **incorrect reverse proxy setup.** While checking subsequent XHR requests against our ELK setup, I noticed that the `POST logstash-*` request to the end of your flow appeared as `text` type instead of `xhr`, with the response body of `ok`. Now, this looks exactly the same as **a successful response from Logstash** when sending in some log events. Our reverse proxy setup consists currently of 2 endpoints handled by [nginx-ingress](https://github.com/kubernetes/ingress-nginx) controller [1] described with the following Kubernetes ingress config: ```auto spec: rules: - host: domain http: paths: - backend: serviceName: logstash servicePort: 5000 path: /log - backend: serviceName: kibana servicePort: 5601 path: /path/to/dashboard/ ``` This translates to the following nginx configuration: ```auto location ~* /log { # proxy settings ... rewrite /log/(.*) /$1 break; rewrite /log / break; proxy_pass http://default-logstash-5000; } location ~* /path/to/dashboard/ { # proxy settings ... rewrite /path/to/dashboard/(.*) /$1 break; rewrite /path/to/dashboard/ / break; proxy_pass http://default-kibana-5601; } ``` The exact problem is that `location ~* /log` directive is a **partial match,** which succeeds for: ```auto POST /path/to/dashboard/es_admin/.kibana/index-pattern/logstash-* ``` disturbing the proper request flow, and in effect distorting the user interface. Available solutions are: 1. making sure `location ~* /path/to/dashboard/` match is performed first - however I am not aware of any possibility of enforcing ingress rules' order in `nginx.conf` file, or 2. using `/log$` path instead of `/log` in ingress settings to enforce suffix match for Logstash - still problematic, as it will allow `/anything/here/zzlog` - but **solves the issue,** or 3. enforing exact path match for Logstash with nginx's `location = /log` directive - this is the **proper solution** but as of Nov 14, 2017, [nginx-ingress](https://github.com/kubernetes/ingress-nginx) controller does not support this feature; there is an open [pull request](https://github.com/kubernetes/ingress-nginx/pull/1415) that implements it. **Thank you** @chrisronline and @cjcenizal for your help! [1] [gcr.io/google\_containers/nginx-ingress-controller:0.8.3](http://gcr.io/google_containers/nginx-ingress-controller:0.8.3) ---
**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\ **Post date:** [November 14, 2017, 7:32pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/8 "2017-11-14T19:32:25Z")
@mg6 Great to hear! Very nice debugging there. Reverse proxies are always a tricky thing but I'm glad you were able to resolve the issue. ---
**Author:** ![cjcenizal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cjcenizal/32/11216_2.png) [@cjcenizal](https://discuss.elastic.co/u/cjcenizal)\ **Post date:** [November 14, 2017, 7:53pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/9 "2017-11-14T19:53:49Z")
Awesome! Glad you could sort it out! Nice work @chrisronline. 😄 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 12, 2017, 7:54pm UTC](https://discuss.elastic.co/t/courier-fetch-error-when-creating-index-pattern-5-5-1/107038/10 "2017-12-12T19:54:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.