# Filebeat hitting payload size limit with AWS ElasticSearch

**URL:** <https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 27, 2018, 6:09pm UTC](https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396 "2018-11-27T18:09:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![spiffytech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiffytech/32/28062_2.png) [@spiffytech](https://discuss.elastic.co/u/spiffytech)\
**Post date:** [November 27, 2018, 6:09pm UTC](https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396/1 "2018-11-27T18:09:19Z")

</div>

ElasticSearch is telling FileBeat that the payload is too large. I've tried setting `bulk_max_size` to `1` and still have the problem. My logs _shouldn't_ be over 10MB in a single message, but maybe I'm wrong about that.

Here's the output config for my Kubernetes log collector:

```auto
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "kubernetes-%{[beat.version]}-%{+yyyy.MM.dd}"
      bulk_max_size: 1

```

And here's the error from Filebeat:

```auto
INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.2.3
INFO template/load.go:129 Template already exists and will not be overwritten.
INFO pipeline/output.go:105 Connection to backoff(elasticsearch(https://my-elasticsearch.es.amazonaws.com:443)) established
INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
INFO [publish] pipeline/retry.go:191 done
ERROR elasticsearch/client.go:317 Failed to perform any bulk index operations: 413 Request Entity Too Large: {"Message":"Request size exceeded 10485760 bytes"}
INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
INFO [publish] pipeline/retry.go:168 done
ERROR pipeline/output.go:121 Failed to publish events: 413 Request Entity Too Large: {"Message":"Request size exceeded 10485760 bytes"}
INFO pipeline/output.go:95 Connecting to backoff(elasticsearch(https://my-elasticsearch.es.amazonaws.com:443))
INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
INFO [publish] pipeline/retry.go:191 done
INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
INFO [publish] pipeline/retry.go:168 done

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 27, 2018, 7:11pm UTC](https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396/2 "2018-11-27T19:11:02Z")

</div>

Logstash requires a special output plugin to communicate with AWS ES as it does not support HTTP basic auth if I recall correctly. I am therefore not sure Filebeat can index directly into AWS ES, so you may need to send data via Logstash.

---

<div class="post-metadata">

**Author:** ![spiffytech](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spiffytech/32/28062_2.png) [@spiffytech](https://discuss.elastic.co/u/spiffytech)\
**Post date:** [November 27, 2018, 8:17pm UTC](https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396/3 "2018-11-27T20:17:26Z")

</div>

Some data is correctly reaching ElasticSearch directly from Filebeat, so that much does work. But I do still get this error, suggesting not all data is reaching ElasticSearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 25, 2018, 8:17pm UTC](https://discuss.elastic.co/t/filebeat-hitting-payload-size-limit-with-aws-elasticsearch/158396/4 "2018-12-25T20:17:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.