Courier fetch: Bad Gateway (6.4.0)

Elastic Stack Kibana
1

Hi there,

i am fairly new to this... and i am stuck... just installed my ELK stack... filebeat seems to working fine... but i moved on to metricbeat i get this error in Kibana when trying for example to select "metricbeat-*" under "discover"

Fatal Error:
Courier fetch: Bad Gateway

Version: 6.4.0
Build: 17929

when i load the
[[Metricbeat System] Overview]
Dashboard it starts to visualise and then switches to the same Fatal Error.

what might be the problem? where can i try to find out more about it?

2

Can you check your Kibana and Elasticsearch logs for anything further?

3

Elasticsearch isn't logging anything when this happens
I tink there is no Kibana log ... I'll have to look for that later today...

4

I found this in the Nginx error log... may that has something to do with it... (I am sure it has)...

2018/08/27 07:08:36 [error] 27876#0: *193227 upstream sent too big header while reading response header from upstream, client: xx.xx.xx.xx, server: xxxx.xx, request: "POST /elasticsearch/_msearch HTTP/1.1", upstream: "http://127.0.0.1:5601/elasticsearch/_msearch", host: "xxxxx.xx:5601", referrer: "http://xxxx.xx:5601/app/kibana"

5

I'm getting that, too. I just updated metricbeat to 6.4 and I'm only getting this from the metricbeat index. I can view other indexes (audit beat, filebeat, etc.) fine.

This makes the second consecutive metricbeat update that's broken my system...I think I'm going to stop updating for a while until things settle down with it.

1 Like
6

I found something about the "upstream sent too big header"
i had to add

    proxy_buffer_size          128k;
    proxy_buffers              4 256k;
    proxy_busy_buffers_size    256k;

to my nginx configuration. that helps.

server {
  listen xxx.xxx.xxx.xxx:5601;

  server_name xxx.de;

  auth_basic "Restricted Access";
  auth_basic_user_file /etc/nginx/htpasswd.users;

  location / {
      proxy_pass http://localhost:5601;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection 'upgrade';
      proxy_set_header Host $host;
      proxy_cache_bypass $http_upgrade;
      proxy_buffer_size          128k;
      proxy_buffers              4 256k;
      proxy_busy_buffers_size    256k;
  }
}
7 Likes
7

Its really works! Thanks!

8

Sir, that was it!

9

That was the problem. Thanks a lot

10

This works. Than you!

11

This should be mentioned in the docs!

12

I'm experiencing the same issue 'Courier fetch: Bad Gateway' as well as just with querying my metricbeat index, but I am not using nginx. I am using haproxy. Anyone know how I can set those limits in haproxy?

13

I found how to resolve for those using haproxy. In the global section of your config, add the following:
tune.bufsize 128000

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.