I'm posting this in ECE as the ES clusters live inside ECE, but that may or may not be the issue. Currently working through a ticket but no resolutions yet.
Whenever I built a new ES Cluster 6.3.2 with Kibana enabled, I get "Reticulating Splines.." on the screen in Kibana. This appears in "Management" and "Index Patterns". It just sits there and spins and you can never create an index pattern.
Normal operations in Dev Console are fine. Anyone else seeing this?
Did you get any resolution on this? I had a quick scan through our internal support issues and couldn't find any mention of "reticulating splines" (which google suggests is a "joke" error, just meaning the server is doing something)
I've never seen this before fwiw, does the HTTP traffic (ie seen via the network console) give any clues? My guess is that it's waiting for some response from the server which never comes back, but knowing what the request is would help narrow it down
Good call. I do see errors in chrome\developertools\console. (replacing our private dns name space for confidentiality.)
INFO: 2018-08-07T15:16:42Z
Adding connection to https://b0bad4b757d24dbf98d388873b32e80d.ip.my.domain.com:9243/elasticsearch
:9243/bundles/kibana.bundle.js:7 setting active api to [es_5_0]
:9243/elasticsearch//_search:1 Failed to load resource: the server responded with a status of 403 ()
:9243/bundles/vendors.bundle.js:3 Promise: Detected an unhandled Promise rejection.
Authorization Exception :: {"path":"//_search","query":{},"body":"{"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}","statusCode":403,"response":"\r\n403 Forbidden\r\n<body bgcolor="white">\r\n
Interesting, 403 Forbidden means that the authenticated user does not have authorization to call _search
Are you logging in as elastic or a different user?
logged in as the elastic user.
And if you do something like
curl -k -u elastic:YOUR_PASSWORD -H 'Content-type: application/json' https://ES_CLUSTER_ID.ip.my.domain.com:9243/_search -d '{"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}', which I think is the exact query that Kibana is failing on, what does it return?
Also probably worth checking out the Kibana logs (eg search for the Kibana cluster id in the logging and monitoring cluster), or they are on disk in /mnt/data/elastic/RUNNER_ID/services/allocator/container/kibana/KIBANA_ID/logs, in case there are some errors before that which give a clue as to why elastic is failing in this way
curl -u 'elastic:blah' -H 'Content-type: application/json' https://myclusterid.ip.es.mydomain.com:9243/_search -d {"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}
{"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unrecognized token 'size': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@722053b8; line: 1, column: 6]"}],"type":"json_parse_exception","reason":"Unrecognized token 'size': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@722053b8; line: 1, column: 6]"},"status":500}curl: (3) [globbing] nested brace in column 15
curl: (3) [globbing] nested brace in column 15
did you miss `` (single quotes) around the JSON object following -d?
○ curl -u 'elastic:blah' -H 'Content-type: application/json' https://mycluster.ip.es.mydomain.com:9243/_search -d '{"size":0,"aggs":{"indices":{"terms":{"field":"_index","size":200}}}}'
{"took":12,"timed_out":false,"_shards":{"total":53,"successful":53,"skipped":0,"failed":0},"hits":{"total":114305,"max_score":0.0,"hits":[]},"aggregations":{"indices":{"doc_count_error_upper_bound":0,"sum_other_doc_count":0,"buckets":[{"key":"metrics-20180805","doc_count":15840},{"key":"metrics-20180807","doc_count":15837},{"key":"metrics-20180804","doc_count":15836},{"key":"metrics-20180806","doc_count":15754},{"key":"metrics-20180803","doc_count":15743},{"key":"metrics-20180802","doc_count":15035},{"key":"metrics-20180808","doc_count":11979},{"key":"metrics-20180801","doc_count":2472},{"key":"items","doc_count":770},{"key":"builds-20180730","doc_count":520},{"key":"builds-20180716","doc_count":403},{"key":"builds-20180726","doc_count":306},{"key":"builds-20180718","doc_count":256},{"key":"builds-20180725","doc_count":247},{"key":"builds-20180806","doc_count":246},{"key":"builds-20180724","doc_count":232},{"key":"builds-20180713","doc_count":198},{"key":"builds-20180719","doc_count":198},{"key":"builds-20180802","doc_count":192},{"key":"builds-20180801","doc_count":186},{"key":"builds-20180717","doc_count":168},{"key":"builds-20180807","doc_count":166},{"key":"builds-20180720","doc_count":165},{"key":"builds-20180727","doc_count":150},{"key":"builds-20180731","doc_count":139},{"key":"builds-20180803","doc_count":134},{"key":"builds-20180723","doc_count":121},{"key":"builds-20180808","doc_count":117},{"key":"nodes-20180807","doc_count":116},{"key":"nodes-20180806","doc_count":98},{"key":"nodes-20180808","doc_count":97},{"key":"builds-20180712","doc_count":80},{"key":"nodes-20180803","doc_count":78},{"key":"nodes-20180802","doc_count":75},{"key":"builds-20180729","doc_count":52},{"key":"builds-20180728","doc_count":42},{"key":"builds-20180722","doc_count":39},{"key":"builds-20180721","doc_count":38},{"key":"nodes-20180804","doc_count":36},{"key":"nodes-20180801","doc_count":35},{"key":"builds-20180804","doc_count":28},{"key":"nodes-20180805","doc_count":27},{"key":"builds-20180715","doc_count":20},{"key":"builds-20180714","doc_count":18},{"key":"builds-20180805","doc_count":14},{"key":".kibana","doc_count":2}]}}}%
this works.
That's really strange ... actually if you look in the proxy logs in the logging and monitoring cluster you should see the 403ing request and be able to tell what the URL was and (I think!) what the calling user was
Hmm. Well I just found out something interesting. In my ECE PREPROD environment, this happens on "any" flavor of 6.3 cluster. Any version, 6.3.1 6.3.2. In my "PROD" ECE environment, this does not happen. So that tells me there's something up with a configuration somewhere. Okay so now where do I look. aws elb config, security groups, where...
Can you give more details on what your ECE PREPROD looks like? I can't immediately see what AWS setting could cause this. I think the thing to do is check the following logs:
(feel free to (re?) raise this as a ticket incidentally, and I'll update this thread with the TLDR once we figure out what's going on)
Problem Found. After searching the proxy logs for "403"s and not finding them, I turned my focus to the aws elb. That was the issue. There was an AWS Waf rule tied to the elb that was generating the 403's. I highly recommend using a tool called "Insomnia" as I was able to see a header presenting the 403's which clearly showed me the aws load balancer was causing the issue.
@kcurtis - glad you got it working and thanks for sharing the resolution - really appreciate it
Alex
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.