Filebeat hitting payload size limit with AWS ElasticSearch

[spiffytech]

ElasticSearch is telling FileBeat that the payload is too large. I've tried setting bulk_max_size to 1 and still have the problem. My logs shouldn't be over 10MB in a single message, but maybe I'm wrong about that.

Here's the output config for my Kubernetes log collector:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "kubernetes-%{[beat.version]}-%{+yyyy.MM.dd}"
      bulk_max_size: 1

And here's the error from Filebeat:

INFO    elasticsearch/client.go:712     Connected to Elasticsearch version 6.2.3
INFO    template/load.go:129    Template already exists and will not be overwritten.
INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(https://my-elasticsearch.es.amazonaws.com:443)) established
INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
INFO    [publish]       pipeline/retry.go:191     done
ERROR   elasticsearch/client.go:317     Failed to perform any bulk index operations: 413 Request Entity Too Large: {"Message":"Request size exceeded 10485760 bytes"}
INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
INFO    [publish]       pipeline/retry.go:168     done
ERROR   pipeline/output.go:121  Failed to publish events: 413 Request Entity Too Large: {"Message":"Request size exceeded 10485760 bytes"}
INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(https://my-elasticsearch.es.amazonaws.com:443))
INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
INFO    [publish]       pipeline/retry.go:191     done
INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
INFO    [publish]       pipeline/retry.go:168     done

[Christian_Dahlqvist]

Logstash requires a special output plugin to communicate with AWS ES as it does not support HTTP basic auth if I recall correctly. I am therefore not sure Filebeat can index directly into AWS ES, so you may need to send data via Logstash.

[spiffytech]

Some data is correctly reaching ElasticSearch directly from Filebeat, so that much does work. But I do still get this error, suggesting not all data is reaching ElasticSearch.