Error: expected 'location' header to be set

Elastic Stack Kibana
1

Hello, I have an issue in connecting Kibana (7.8.0 OSS) running on my local machine to our Elasticsearch (7.8.0) instance on our Kubernates cluster.

The initial connection works and Kibana creates the index .kibana_1 on Elasticsearch. However, when I try to run a search, it fails with a 500 error with following message:

      log   [13:52:56.588] [error][http] Error: expected 'location' header to be set
        at HapiResponseAdapter.toRedirect (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:112:13)
        at HapiResponseAdapter.toHapiResponse (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:94:19)
        at HapiResponseAdapter.handle (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:81:17)
        at Router.handle (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/router.js:160:34)
        at process._tickCallback (internal/process/next_tick.js:68:7)
     error  [13:52:56.429]  Error: Internal Server Error
        at HapiResponseAdapter.toInternalError (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/response_adapter.js:69:19)
        at Router.handle (/Users/ttakamiy@us.ibm.com/kibana/kibana-7.8.0-darwin-x86_64/src/core/server/http/router/router.js:163:34)
        at process._tickCallback (internal/process/next_tick.js:68:7)

Can anybody see anything that potentially causes this kind of errors? Thank you!

Tami

2

Hi @TamiTakamiya

How are you running this search that is causing this error to happen?

3

It can be caused by clicking one of my search on Kibana screen. For example, running the GET / can cause the error.

image
image2110×886 106 KB

4

Can you click the "Wrench" icon, select "Copy as cURL", and then try pasting it into your terminal and running it from there? The cURL request should be something like this:

curl -XGET "http://localhost:9200/"

I'm trying to narrow down whether this is an issue within Console or a problem with the way Kibana is interacting with the Kubernetes cluster. Thanks.

5

"Wrench" icon gave me the correct URL without user/password. By providing the user/password with curl's -p option, I could run the generated curl command successfully.

6

I still do not fully understand how this error occurs, but could find a workaround.

When the error occurs, my local Kibana instance seemed to sends to send a request to Elasticsearch with following headers:

x-forwarded-for: 127.0.0.1
x-forwarded-port: 52048
x-forwarded-proto: http
x-forwarded-host: localhost:5601

Among them x-forwarded-proto: http is the one causing this. If I send a valid Elasticsearch query with it using curl, I got:

<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

So I guess our nginx (Kubernetes Ingress) is configured in that way.

My workaround is to force Kibana not to generate those headers, i.e. edit kibana-7.8.0-darwin-x86_64/src/plugins/console/server/routes/api/console/proxy/create_handler.js and comment out following lines:

  if ((_req === null || _req === void 0 ? void 0 : (_req$info = _req.info) === null || _req$info === void 0 ? void 0 : _req$info.remotePort) && (_req === null || _req === void 0 ? void 0 : (_req$info2 = _req.info) === null || _req$info2 === void 0 ? void 0 : _req$info2.remoteAddress)) {
    // see https://git.io/vytQ7
    extendCommaList(headers, 'x-forwarded-for', _req.info.remoteAddress);
    extendCommaList(headers, 'x-forwarded-port', _req.info.remotePort);
    extendCommaList(headers, 'x-forwarded-proto', _req.server.info.protocol);
    extendCommaList(headers, 'x-forwarded-host', _req.info.host);
  }

in the function getProxyHeaders. Although there would be a cleaner fix, Kibana OSS 7.8.0 works for me with that modification.

Tami

7

Thanks for the information, Tami! I don't know a lot about nginx, but I wonder if this is something you can solve by changing its configuration.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.