Will Elasticsearch reject documents based on document size?


(Amruth) #1

Hi,

Will Elasticsearch reject or drop documents if document size is too large? I am ingesting documents using logstash but I don’t see any error logs in Logstash or Elasticsearch.

How do I know if Elasticsearch rejects any documents?


(swarmee.net) #2

Hey Amruth, simple answer -- if documents are being rejected you should get messages in the logs.

Root cause - if you have "missing" documents in elasticsearch its generally cause of mapping conflicts. E.g. you are POSTing in a document that has text in a date field.

If you do have super large fields you can set the ignore above parameter for your text fields - so it will not fail when the field is to large - as explained here --> https://www.elastic.co/guide/en/elasticsearch/reference/current/ignore-above.html

The best thing to do is to just try to manually post the missing reports into elastic using the dev_tools tab in Kibana or via curl. If it failed you will get a pretty good error message that will lead you down the right track.


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.