Kibana 4 authorization exception error

Elastic Stack Elasticsearch
1

We use AWS elasticsearch service and kibana is installed on the ec2 instance. From last few days, we started getting below error.
Could you please let us know what may cause this error?

############
Fatal Error
Kibana: Unknown error while connecting to Elasticsearch
Error: Unknown error while connecting to Elasticsearch
Error: Authorization Exception
at respond (http://<IP_Address>:9300/index.js?_b=7616:86367:15 1)
at checkRespForFailure (http://<IP_Address>:9300/index.js?_b=7616:86335:7)
at http://<IP_Address>:9300/index.js?_b=7616:84973:7
at wrappedErrback (http://<IP_Address>:9300/index.js?_b=7616:20902:78)
at wrappedErrback (http://<IP_Address>:9300/index.js?_b=7616:20902:78)
at wrappedErrback (http://<IP_Address>:9300/index.js?_b=7616:20902:78)
at http://<IP_Address>:9300/index.js?_b=7616:21035:76
at Scope.$eval (http://<IP_Address>:9300/index.js?_b=7616:22022:28)
at Scope.$digest (http://<IP_Address>:9300/index.js?_b=7616:21834:31)
at Scope.$apply (http://<IP_Address>:9300/index.js?_b=7616:22126:24)
##########

The aws elasticsearch service is up and running. Here is the cluster health status:

{"cluster_name":"599427016059:mwanalytics","status":"green","timed_out":false,"number_of_nodes":9,"number_of_data_nodes":6,"active_primary_shards":1626,"active_shards":3252,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"number_of_pending_tasks":0}

Could you please let us know what could be the cause of this error?

Thanks,
Santosh

2

This looks wrong http://<IP_Address>:9300. It should be http://<IP_Address>:9200 or whatever REST post AWS exposes...

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, APM, Logs UI, Infra UI, SIEM, Maps UI and what is coming next :slight_smile: ...

3

Thnaks @dadoonet for your reply.
The result is same with port 9200.

4

Does it mean that you modified manually the logs you pasted initially and that I should have read 9200 instead of 9300? As I don't have a clear view on what you are exactly doing it's hard to answer.

But one comment though: You have 3252 shards on 6 data nodes. Which means 542 shards per node. So I assume that your nodes have more than 28 gb of HEAP size. Is that true?
What are the elasticsearch logs please?

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

image
image.jpg764×92 16.4 KB

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.

5

Hi @dadoonet
It is 9300 as stated in my 1st post.
I changed the port to 9200 and result is same (same authorization exception error).
We are using AWS elasticsearch service so elasticsearch logs aren't available.

here is our architecture:
Our source application sends data to nginx, nginx is configured to send data to AWS elasticsearch service. Kibana is deployed on the same server where nginx is hosted.

The kibana used to work earlier. Nothing changed at nginx/kibana server but kibana started throwing the error I mentioned earlier.

6

What are you using nginx for?

Nothing changed at nginx/kibana server but kibana started throwing the error I mentioned earlier.

I have no clue.

May be contact AWS support then. I'm not sure we can help.

It is 9300 as stated in my 1st post.

I thought that AWS never allowed this port to be exposed but may be that changed.

I changed the port to 9200 and result is same (same authorization exception error).

What are the full logs in that case?

Our source application sends data to nginx, nginx is configured to send data to AWS elasticsearch service. Kibana is deployed on the same server where nginx is hosted.

So Kibana is directly connected to Elasticsearch right? Not to nginx... Am I correct?
If so and if the logs I'm seeing are really coming from Kibana, I don't understand how this has been working in the past and is suddenly failing as you wrote here:

The kibana used to work earlier. Nothing changed at nginx/kibana server but kibana started throwing the error I mentioned earlier.

Kibana has never been able to communicate using the Transport Protocol AFAIK.

So something is really unclear to me.
BTW do you mean that Kibana is not provided by AWS out of the box and that you are managing it manually by yourself? Are you sure that your elasticsearch instances have not been upgraded to a new version recently and that your Kibana is not up to date anymore?
I don't know how security works with AWS but may be it requires a specific plugin in Kibana?

Again, I'd really recommend looking at the official service where everything is setup for you, up and running (including Kibana, all behind a proxy) in few minutes. You don't even need a credit card to test it from cloud.elastic.co as you have 14 days for free... :slight_smile:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.