How to authenticate Docker kibana instance to AWS ES

Elastic Stack Kibana
1

I am facing an issue to connect my Docker Kibana Instance to AWS ES. The error is Authorization exception. I can curl from the EC2 host and within the container. I have included the IP of the client to the access policy. I am very new to ELK so would be awesome if somebody can explain how to achieve this in details.

My guess is that you might need to provide AWS access key inside the container and use boto3 to sign the requests even for Kibana index? I tried providing the ES url as an ENV varibale to docker compose and ES access policy allows the Docker host IP, but that did not solve the issue. Also I read from few sources that you can provide ES name and passowrd in kibana.yml if that is possible would anyone know what the default values for the AWS ES?

I am trying the Docker Kibana as the ES provided Kibana instance is broken and I could not figure out why and limited documentation from AWS. It looks like logstash can create index on ES but if I try to create an index pattern on Kibana it just gives me error and the page is broken.

1 Like
2

Why are you looking to run external Kibana instead of the one provided by AWS?

3

Because something is broken on the Kibana side of ES. I can see that there is index in the ES. I created index patterns. But when I try to Discover or Visualize tab then it complains I must choose a default index pattern. Fair enough, I tried clicking one of the two index patterns i created to use it as default. Kibana loads for few seconds and nothing happens. Please find screenshot for the current Kibana state.https://github.com/MerrilCode/b2b_readme_images/blob/master/Screenshot%202019-06-12%20at%2011.09.45.png

Also when I create a new index pattern then I get bunch of errors and it does not really create the index patternhttps://github.com/MerrilCode/b2b_readme_images/blob/master/Screenshot%202019-06-12%20at%2011.18.44.png

4

Which docker image are you using? How are you configuring Kibana?

5

Here is my Docker file

FROM docker.elastic.co/kibana/kibana:6.7.0
RUN mkdir ~/.aws
ADD ./config /$HOME/.aws
ENV ELASTICSEARCH_URL=http://domain_name.eu-west- 
2.es.amazonaws.com:80
EXPOSE 5601
CMD ["/bin/bash"]
VOLUME ["/usr/share/kibana/config"]

and my docker-compose.yml

version: '3'
services:
    kibana:
       image: kibana_6.7.0
       ports:
             - "5601:5601"
       environment:
       SERVER_NAME: kibana-test
        ELASTICSEARCH_HOSTS: http://domain_name.eu-west-2.es.amazonaws.com:80

ignore any indentation on the files

I tried to use just a custom created image from Dockerfile and the official Kibana image.

I am not sure how to configure kibana to be honest. I thought providing a kibana.yml under $KIBANA_HOME/config with ES domain url was good enough when docker runs the container. I have opened the port on aws for the kibana but the site cannot be reached.

here is my kibana.yml

server.port: 5601
elasticsearch.hosts: ["http://domain_name.eu-west- 
2.es.amazonaws.com:80"]
kibana.index: ".kibana-5"
6

You probably need to use the OSS image of Kibana as AWS ES does not support X-pack, which I believe the default distribution checks for.

7

Thanks Christian. I will try that. Any idea why the managed Kibana does not work on AWS ES?

8

Now I can see Kibana UI. But the issue is still there or even worse. Kibana is connected to AWS ES domain because I can see the index from Dev Tools
[https://github.com/MerrilCode/b2b_readme_images/blob/master/Screenshot%202019-06-12%20at%2012.27.15.png](Dev Tools screenshot)

But if I try to set an index pattern or select and index pattern already present then I get a blank screen. [https://github.com/MerrilCode/b2b_readme_images/blob/master/Screenshot%202019-06-12%20at%2012.27.34.png](Index pattern screenshot)

I am starting to wonder if it is some sort of plugin issue or permission issue. I tried resetting the default index from Advanced settings but I am getting a request 503 error. If it is related to permission issue could you direct me to a relevant documentation?

Many thanks,
Merril.

9

No idea at all. You need to contact AWS support for that.

10

Actually I am getting the same error as before with kiban-oss image. "User: anonymous is not authorized to perform: es:ESHttpPut\"}"} I forgot to change the elasticsearch.url in kibana.yml before. back to the same issue now. It is definitely AWS authentication error but how do I authenticate within a container?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.