Discover fails with "content length bigger than max allowed string" on index with large text field

Elastic Stack Elasticsearch
1

Hello Connections,

We're facing an issue when trying to retrieve documents in Discover, and wanted to get community input on the best long-term fix.

Environment:

  • Elastic Stack version: [8.19.3]
  • Deployment: self-managed on EC2
  • Ingestion path: Filebeat → Logstash → Elasticsearch

The error:

When loading Discover against one of our production data views (index-* indices), we get:

942f5b_image
942f5b_image576×285 9.21 KB

Root cause we've identified:

One field in our documents (a payload/raw_payload text field, is extremely large on a subset of documents — we found individual documents as large as ~23MB in a single index, versus ~2KB for typical documents in the same index. When Discover's default query returns full _source for a batch of matched hits, the aggregate response for even a short time window (minutes, not hours) exceeds ~512MB and the browser fails to parse it.

We need to have the document and where the payload is in event.original

Any suggestion to fix

Thanks in advance

2

welcome to the forum @Mouly_Infy

This has been asked directly or indirectly a couplee of times, e.g. here

See this github thread too for some ideas.

You need be careful here too as other clients (not just Kibana/Discover) can hit issues with large HTTP payloads. I dont know your use case and what you are specifically trying to do, but it's unlikely there is much value in "seeing" a 23 MB document within Kibana.