Coworker of Raman, following up with our solution to the problem for the benefit of future generations.
The issue ended up being Kubernetes related. Our elasticsearch stack is running in Kubernetes, and we were using some fairly bad defaults values for our nginx ingress controller configurations.
The indicator ended up being fairly obvious, the filebeat log even prints out the error message complete with a mention of nginx in its error.
Due to the default settings of the nginx ingress controller, nginx itself would reject any payloads larger than 10kb. Also, the ingress controllers would frequently die due to only having 64 MB of memory available.
Elasticsearch itself could handle the bulk index requests with no issue, but requests never made it there because nginx would reject them before it could get that far.
The fix was to reconfigure the nginx ingress to accept larger payloads than the default.
I'm not the best at k8s patching, but I'll paste the commands I ran to fix my k8s cluster, which then resolved the issue from the filebeat side and got our logs ingested.
kubectl -n ingress-nginx get configmap ingress-internal-nginx-ingress-controller -o yaml | sed '/^data:.*/a \ \ proxy-body-size: \"10m\"' > /tmp/tempConfigMap; kubectl apply -f /tmp/tempConfigMap; rm /tmp/tempConfigMap
This pipes the nginx ingress' configmap, adds the proxy-body-size configuration value (set to 10m), writes that to a temp file, applies that temp file, then deletes the temp file.
kubectl -n ingress-nginx patch deployment ingress-internal-nginx-ingress-controller --patch "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"nginx-ingress-controller\",\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"512Mi\"},\"requests\":{\"cpu\":\"1\",\"memory\":\"256Mi\"}}}]}}}}"
This patches the nginx ingress deployment to include some sane default values for resource requests and limits. This is very off-the-cuff, and not at all based on specific usage patterns - your cluster may differ from mine, and your nginx pods might need more memory/cpu (or less!).
Added bonus, patching the nginx ingress deployment restarts all of the nginx ingress pods, which also pulls up the new configmap.