Kibana Region Maps in AWS EC2

[georgezoto]

Hello Kibana team,

This might be a question right on the boundary of Kibana and AWS but I am not able to find any guidance online. I am hosting Elasticsearch and Kibana 6.2.4 on a 3 node EC2 cluster for the whole purpose of utilizing region maps and forecasting (elastic cloud does not offer neither at the moment ).

I am hosting all my custom region maps as geojson files in a public S3 bucket.
Looking at the network traffic leaving the Kibana - EC2 instance I see requests towards the public bucket.

I enabled access logging in this public S3 bucket and I do not see Kibana's requests reaching it.
As a result I do not see any map output on my region map visualization

Can anyone guide me and help me pinpoint the issue? Could it be S3 policy related?

For this cluster I have very limited inbound access policies in place (port 9200 for ES API, port 9300 for internal ES communication and port 9047 for Dremio) and all traffic for outbound (which should be the case for Kibana region map requests accessing the external S3 bucket).

Do I need a specific rule on the inbound as well? What is the most secure way to accomplish this?

Thank you in advance,
George Zoto

[spalger (Spencer Alger)]

Have you checked the network tab in the dev tools? Are there any logs in the browser that indicate if the requests are hanging?

Hanging requests is generally a sign to me that AWS is blocking the request somehow.

[georgezoto]

Awesome start to troubleshoot this issue @spalger!
For some reason in the dev tools network tab I see a 403 Forbidden response code.
I am able to access the same URL on my browser though.

1. Is this related to the kbn-verion headers sent by Kibana?
2. Does CORS have to be in the picture here?

Thank you again,
George

[spalger (Spencer Alger)]

Yes, I suppose it's possible that the kbn-version header is annoying s3 and it is responding with a 403 because it doesn't want to get that header... let me see if I can find a way to work around this without proxying requests to s3

[spalger (Spencer Alger)]

Unfortunately I've confirmed that there isn't currently any way to prevent Kibana from sending this header on requests for the shape files. Perhaps you need to setup a reverse proxy that strips the kbn-version header from those request before sending them... Maybe you could serve them from your Kibana instances with a kibana plugin that proxies the requests to s3...

[spalger (Spencer Alger)]

Looks like you might be able to configure your s3 bucket to accept the kbn-version header: https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html

[thomasneirynck]

@georgezoto,

can you modify the CORS settings for your S3 bucket to explicitly allow the kbn-version header?

https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html

We've had issues in the past that some browsers/server-combos do not handle the * wildcard correctly. This was not for S3 in particular, but may be an issue here.

[nickpeihl]

I can confirm that the default CORS settings for public S3 buckets will not work with the kbn-version header. You specifically have to allow this in the CORS setting for the bucket.

For example, here are the CORS settings that worked for my test S3 bucket.

<CORSConfiguration>
	<CORSRule>
		<AllowedOrigin>*</AllowedOrigin>
		<AllowedMethod>GET</AllowedMethod>
		<MaxAgeSeconds>3000</MaxAgeSeconds>
		<AllowedHeader>kbn-version</AllowedHeader>
	</CORSRule>
</CORSConfiguration>

[georgezoto]

Finally, this is working . Thank you @spalger, @thomasneirynck and @nickpeihl, S3 CORS configuration was successful.

For completeness here is an extra link I used: https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-cors-configuration.html

Thank you,
George