# Fix to libbeats to split bulk requests which are too large, not working in Elastic Agent 8.9.0?

**URL:** <https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136>\
**Category:** Beats\
**Tags:** libbeat\
**Created:** [September 1, 2023, 4:10pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136 "2023-09-01T16:10:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Craig\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/craig_rodrigues/32/121875_2.png) [@Craig\_Rodrigues](https://discuss.elastic.co/u/Craig_Rodrigues)\
**Post date:** [September 1, 2023, 4:10pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/1 "2023-09-01T16:10:18Z")

</div>

In Beats, I see that this commit was merged in March 2023:

" Split large batches on error instead of dropping them"  
[PR 34911](https://github.com/elastic/beats/pull/34911)

I think that [PR 34911](https://github.com/elastic/beats/pull/34911) changed the `Publish()` logic:

If I look here:

> <https://github.com/elastic/beats/blob/1a7e9aa9819073e9be5112310d9d419ca1458710/libbeat/outputs/elasticsearch/client.go#L187>

```auto
func (client *Client) Publish(ctx context.Context, batch publisher.Batch) error {
	events := batch.Events()
	rest, err := client.publishEvents(ctx, events)

	switch {
	case errors.Is(err, errPayloadTooLarge):
		if batch.SplitRetry() {
			// Report that we split a batch
			client.observer.Split()
		} else {
			// If the batch could not be split, there is no option left but
			// to drop it and log the error state.
			batch.Drop()
			client.observer.Dropped(len(events))
			err := apm.CaptureError(ctx, fmt.Errorf("failed to perform bulk index operation: %w", err))
			err.Send()
			client.log.Error(err)
		}
		// Returning an error from Publish forces a client close / reconnect,
		// so don't pass this error through since it doesn't indicate anything
		// wrong with the connection.
		return nil
	case len(rest) == 0:
		batch.ACK()
	default:
		batch.RetryEvents(rest)
	}
	return err
}

```

So if my understanding of the logic is correct,  
this function should try to split up a bulk request  
and only fail if it cannot successfully split up the bulk request and send.

I am running elastic-agent 8.9.0 , and I do not seem  
to see this behavior. I am getting hundreds of errors like the one I posted here:

> [@Elastic-agent is triggering HTTP 413 (entity too large) errors](https://discuss.elastic.co/t/elastic-agent-is-triggering-http-413-entity-too-large-errors/341932):
>
> I have an elastic agent: elastic-agent version Binary: 8.9.0 (build: dc443bc2427920a26141b05f9c07a52191881af5 at 2023-07-19 20:55:16 +0000 UTC) Daemon: 8.9.0 (build: dc443bc2427920a26141b05f9c07a52191881af5 at 2023-07-19 20:55:16 +0000 UTC) elastic-agent status ┌─ fleet │ └─ status: (HEALTHY) Connected └─ elastic-agent └─ status: (HEALTHY) Running If I do: elastic-agent logs | jq I am seeing some errors like: { "log.level": "error", "@timestamp": "2023-08-29T22:51:52.377Z", "me…

A few questions, maybe @faec @Lee_Hinman @cmacknz can help:

1. Does elastic-agent 8.9.0 have the fix in [PR 34911](https://github.com/elastic/beats/pull/34911) ?
2. Is this fix working properly?
3. How do I tell how large the payload that libbeat is trying to send to elasticsearch via a `POST _bulk` API call?
4. Does libbeat ignore the setting of `http.max_content_length` on the server?

Any help you can give to help shed light on this,  
and solve my problem would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![cmacknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmacknz/32/103773_2.png) [@cmacknz](https://discuss.elastic.co/u/cmacknz)\
**Post date:** [September 6, 2023, 8:33pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/2 "2023-09-06T20:33:04Z")

</div>

> 1. Does elastic-agent 8.9.0 have the fix in [PR 34911](https://github.com/elastic/beats/pull/34911) ?

Yes it does, you can see the tags that include it on the merge commit [Split large batches on error instead of dropping them (#34911) · elastic/beats@df59745 · GitHub](https://github.com/elastic/beats/commit/df59745ee01a9ffd86387160c0d63a7a4b5d8f5c)

> Is this fix working properly?

It doesn't appear to be having any effect here, and from what you've posted I suspect the reason might be that Elastic Agent isn't actually seeing the HTTP 413 response and is instead seeing `write tcp [redacted]:33430->[redacted]:443: write: broken pipe` as the error. The new code won't do anything unless it explicitly receives a 413 from Elasticsearch.

> 1. How do I tell how large the payload that libbeat is trying to send to elasticsearch via a `POST _bulk` API call?

Ideally by getting a 413 response back

> 1. Does libbeat ignore the setting of `http.max_content_length` on the server?

Yes because it doesn't know about it. Libbeat does not query this parameter and Elasticsearch does not give it back.

---

<div class="post-metadata">

**Author:** ![Craig\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/craig_rodrigues/32/121875_2.png) [@Craig\_Rodrigues](https://discuss.elastic.co/u/Craig_Rodrigues)\
**Post date:** [September 7, 2023, 6:30pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/3 "2023-09-07T18:30:09Z")

</div>

Are you sure that Elasticsearch does not give this parameter back?

If I do:

[GET \_cluster/settings?flat\_settings=true&include\_defaults=true]( [Cluster get settings API | Elasticsearch Guide [8.9] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.9/cluster-get-settings.html)

I see a bunch of setings, including:

```auto
 "http.max_content_length": "100mb",

```

If this parameter is queryable from Elasticsearch, would it be possible to  
change elastic-agent / beats to query this parameter, and adjust the size of the  
payload being sent by `POST _bulk` calls?

---

<div class="post-metadata">

**Author:** ![cmacknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmacknz/32/103773_2.png) [@cmacknz](https://discuss.elastic.co/u/cmacknz)\
**Post date:** [September 7, 2023, 6:45pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/4 "2023-09-07T18:45:16Z")

</div>

Yes sorry, I didn't mean it was impossible to query I meant it wasn't part of the \_bulk response. I suppose we could query it but the batch splitting implementation was supposed to make that unnecessary. If we don't always get 413 responses back when a batch is too large it would make sense for us to do that.

---

<div class="post-metadata">

**Author:** ![Craig\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/craig_rodrigues/32/121875_2.png) [@Craig\_Rodrigues](https://discuss.elastic.co/u/Craig_Rodrigues)\
**Post date:** [September 7, 2023, 7:21pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/5 "2023-09-07T19:21:25Z")

</div>

No worries!  
Should I submit an enhancement request for elastic-agent / beats to query http.max\_content\_length from Elasticsearch so that this ca be used as part of the batch splitting implementation?  
Either somewhere in GitHub, or via [https://support.elastic.co](https://support.elastic.co)?

---

<div class="post-metadata">

**Author:** ![cmacknz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cmacknz/32/103773_2.png) [@cmacknz](https://discuss.elastic.co/u/cmacknz)\
**Post date:** [September 7, 2023, 7:34pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/6 "2023-09-07T19:34:18Z")

</div>

I would do both, create a public GitHub issue in Beats and also raise it through support. Going through support in addition to creating an issue will help with prioritization.

---

<div class="post-metadata">

**Author:** ![Craig\_Rodrigues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/craig_rodrigues/32/121875_2.png) [@Craig\_Rodrigues](https://discuss.elastic.co/u/Craig_Rodrigues)\
**Post date:** [September 8, 2023, 8:45pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/8 "2023-09-08T20:45:22Z")

</div>

I have submitted:

1. [Enhance logic for splitting large payload requests by querying http.max\_content\_length from Elasticsearch · Issue #36534 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/36534)
2. Enhancement Request: #19624

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2023, 10:45pm UTC](https://discuss.elastic.co/t/fix-to-libbeats-to-split-bulk-requests-which-are-too-large-not-working-in-elastic-agent-8-9-0/342136/9 "2023-10-06T22:45:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.