403 Error on Kibaha UI when loading time filter field name

Elastic Stack Kibana
1

HI,
When I try to load the Kibana UI, I see the following error:

    at http://kibana.dev.commandcentral.com/bundles/kibana.bundle.js?v=15571:227:19758
    at processQueue (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:38:23621)
    at http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:38:23888
    at Scope.$eval (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:39:4619)
    at Scope.$digest (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:39:2359)
    at Scope.$apply (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:39:5037)
    at done (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:37:25027)
    at completeRequest (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:37:28702)
    at XMLHttpRequest.xhr.onload (http://kibana.dev.commandcentral.com/bundles/commons.bundle.js?v=15571:37:29634)

Kibana is running on the host: "abc.dev.com" and is configured to use the Elastic Search at "http://elasticsearch-logging:9200" where "elasticsearch-logging" is the name of the service in Kubernetes.

I tried the following:

  1. Allow Ingress to Elastic search
    I assumed that the error is due to CORS where the host names differ, so I created an Ingress to the ES cluster with the name "elasticsearch-logging.abc.com".
    Result: The ES instance running at 9200 fails to find the second instance at 9300. This loops forever and the ES doesn't start.

  2. Renamed the service to "elasticsearch-logging.abc.com"
    Result: The deployment fails because of "." in the service name.

  3. Enable CORS on ES using the following configuration:

http.cors.enabled: true
http.cors.allow-origin: "*"
<Empty Line>

Kibana fails to connect with Elastic Search cluster and keeps looping on an error.
Also, I couldn't use CURL to check the health of ES and indices. I was getting an unauthorized error.

Environment information:
Kibana: 5.6.4 (docker.elastic.co/kibana/kibana:5.6.4)

Elastic Search:5.6.4 (k8s.gcr.io/elasticsearch:v5.6.4)

Default configurations for both Kibana and ES.

PV on Azure Disk

Cluster running on Azure.

2

@challa_sai_kiran are you specifying the username/password when using curl to connect to Elasticsearch? That's a great place to start to ensure the username/password are configured correctly in Elasticsearch before moving on to working on getting Kibana to be able to connect to it. You can use something similar to the following to connect to Elasticsearch and authenticate:

curl -u elastic:changeme http://localhost:9200

Also is that the full error that you're seeing? It appears to have the first few lines missing?

3

I'm able to connect to the ES without the user credentials too (after a port forward)

That is the full error that I see on the UI.

4

@challa_sai_kiran so you aren't using user authentication on Elasticsearch, but you're seeing a 403? Can you open up your browser's dev tools and see which network call is giving you a 403?

5

This is what I got:

Request URL:http://kibana..com/api/index_patterns/_fields_for_wildcard?pattern=logstash-&meta_fields=%5B%22_source%22%2C%22_id%22%2C%22_type%22%2C%22_index%22%2C%22_score%22%5D
Request Method:GET
Status Code:403 ModSecurity Action
Remote Address:52.191.253.189:80
Referrer Policy:no-referrer-when-downgrade
Content-Length:1233
Content-Type:text/html
Date:Mon, 12 Mar 2018 21:06:37 GMT
Server:Microsoft-IIS/10.0
Accept:application/json, text/plain, /
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.9
Connection:keep-alive
Cookie:_ga=GA1.2.38315069.1518475045; ApplicationGatewayAffinity=aec8bf4924904624c014f771c7c7d5052d5ca7537afccc9caa42d74a1f7e0023
Host:kibana..com
kbn-version:5.6.4
Referer:http://kibana..com/app/kibana
User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36
pattern:logstash-
meta_fields:["_source","_id","_type","_index","_score"]

6

@challa_sai_kiran are you running Apache in front of your Elasticsearch instance?

7

There is Nginx in front of Kibana service. Below is the deployment file that I have as of now:

apiVersion: v1
kind: Service
metadata:
  name: kibana-logging
  namespace: monitor
  labels:
    k8s-app: kibana-logging
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "Kibana"
spec:
  ports:
  - port: 5601
    protocol: TCP
    targetPort: ui
  selector:
    k8s-app: kibana-logging
  type: LoadBalancer
---
apiVersion:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kibana-logging
  namespace: monitor
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: kibana.dev.REMOVED.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.