APM server behind nginx closes connection prematurely

[Jorgen_Ader]

The problem: Apm agent starts getting 502 after payload seems to go over ~120kb, have not figured exact value yet.

Http basic auth is used for elastic/kibana/apm (apm agents are ip whitelisted and use token)

Nginx logs

2018/06/13 04:30:12 [error] 4825#4825: *5334 upstream prematurely closed connection while reading response header from upstream, client: *.*.*.*, server: ***, request: "POST /v1/transactions HTTP/1.1"

Apm logs

2018-06-13T04:30:12.941Z	INFO	[request]	beater/handlers.go:150	handled request	{"request_id": "***", "response_code": 202, "method": "POST", "URL": "/v1/transactions", "content_length": 138635, "remote_address": "*.*.*.*", "user_agent": "elasticapm-python/2.1.1"}

relevant Nginx config:

// default nginx config
keepalive_timeout 65;

// upstream server config
upstream apm-server {
    server apm_server:8200;

    keepalive 15;
}

// location block
location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";
        proxy_redirect off;
        proxy_pass http://apm-server;
    }

Apm config

apm-server:
  host: "0.0.0.0:8200"
  secret_token: "the token"
  max_unzipped_size: 104857600
  max_header_size: 10485760
  concurrent_requests: 50

setup.dashboards.enabled: true

setup.kibana.host: "kibana host"
setup.kibana.username: "http user"
setup.kibana.password: "http pw"

output.elasticsearch:
  hosts: ['elastic-host']
  headers:
    Authorization: "Basic x:x"

queue:
  mem:
    events: 16384
    flush.min_events: 512

Versions used:
APM Server: 6.2.4
APM python agent: 2.1.1
Nginx: 1.12.2

[gil (Gil Raphaelli)]

Thanks for the report. This can happen when the apm-server doesn't start writing a response back to the client within the write timeout. In 6.2, you can bump apm-server.write_timeout up to address this - the default in the upcoming release is 30s.