# Filebeat - stalls and effectively hangs when max request size exceeded

**URL:** <https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437>\
**Category:** Beats\
**Created:** [April 25, 2019, 11:18am UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437 "2019-04-25T11:18:24Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kieren\_Johnstone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kieren_johnstone/32/26311_2.png) [@Kieren\_Johnstone](https://discuss.elastic.co/u/Kieren_Johnstone)\
**Post date:** [April 25, 2019, 11:18am UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/1 "2019-04-25T11:18:24Z")

</div>

I'm using filebeat to send raw JSON documents (one per line) to a hosted ([cloud.elastic.co](http://cloud.elastic.co)) ES cluster.

The max request size is not configurable server-side, and despite tweaking the batch size in the elasticsearch config in filebeat, I can't find a good balance.

When filebeat encounters a "request too large" error, it simply retries forever, never successfully sending up anything at all. When I kill the process, it restarts and presumably the batch it prepares is different, and things work again, at least for a time.

Is there some filebeat config I can do to remedy this?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 26, 2019, 11:39am UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/2 "2019-04-26T11:39:20Z")

</div>

Can you share your filebeat configuration and logs? How big is your hosted cluster?

---

<div class="post-metadata">

**Author:** ![Kieren\_Johnstone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kieren_johnstone/32/26311_2.png) [@Kieren\_Johnstone](https://discuss.elastic.co/u/Kieren_Johnstone)\
**Post date:** [April 26, 2019, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/3 "2019-04-26T14:46:40Z")

</div>

```
xpack.monitoring.enabled: true
logging.level: info

cloud.id: "xx:yy"
cloud.auth: "zz"

filebeat.registry_file: D:/home/site/filebeatregistry

filebeat.prospectors:
- type: log
  enabled: true
  json.keys_under_root: true
  json.overwrite_keys: true
  paths:
  - D:/local/temp/jobs/continuous/worker/**/safflog_*.txt
  - D:/home/site/wwwroot/**/safflog_*.txt
  
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml    

setup.template:
  name: "filebeat-rr-logs"
  pattern: "filebeat-rr-logs-*"
  settings:
    index:
      number_of_shards: 1
      number_of_replicas: 0

output.elasticsearch:
  bulk_max_size: 2000
  index: "filebeat-rr-logs-%{[beat.version]}-%{+yyyy.ww}"
  compression_level: 2

```

Logs-wise:

`2019-04-26T08:31:41.672Z	ERROR	elasticsearch/client.go:317	Failed to perform any bulk index operations: 413 Request Entity Too Large:`

(just repeats)

Hosted cluster is 8GB RAM / 192GB storage, single zone.. 1 shard, 0 replicas as above.

Thanks!

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 26, 2019, 4:09pm UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/4 "2019-04-26T16:09:13Z")

</div>

Elasticsearch complains about the size of the bulk request. Have you tried to reduce `bulk_max_size`?

---

<div class="post-metadata">

**Author:** ![Kieren\_Johnstone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kieren_johnstone/32/26311_2.png) [@Kieren\_Johnstone](https://discuss.elastic.co/u/Kieren_Johnstone)\
**Post date:** [May 21, 2019, 8:04am UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/5 "2019-05-21T08:04:12Z")

</div>

Sorry, I didn't see the reply. Yes, but if it's too low, the throughput suffers. It's the occasional very large set of requests that throws things off.

If there was a bulk\_max\_size\_bytes, that would be perfect, of course, but it's a number of items.

The client (filebeat) uses a max number of items, and the server (elasticsearch) use a max request size.. I think that's the issue?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 18, 2019, 10:04am UTC](https://discuss.elastic.co/t/filebeat-stalls-and-effectively-hangs-when-max-request-size-exceeded/178437/6 "2019-06-18T10:04:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.