elasticsearch.exceptions.RequestError: RequestError(400, 'illegal_argument_exception', 'Limit of total fields [1000] in index [pv3] has been exceeded')

[Nikita_Duggal]

I am indexing around 80000 documents with 2 GB of data size from mongodb to elastic search index using Python. After indexing 16000 documents i receive this error.
Note : I have been using elasticsearch locally on my computer and not cloud.

Can you please let me know what exactly can be the issue and how can it be resolved?

[danielmitterdorfer]

Hi,

Elasticsearch has some built-in limits to avoid resource problems and one of them is that you are not allowed to have more than 1000 different fields per document type. The situation you describe sounds as if you index source documents with a vastly different structure into the same index. By default, Elasticsearch will add new fields dynamically as new documents are added and it seems the documents that you ingest are so different that you end up with more than 1000 different fields in one document type which triggers this error.

I suggest that you start by reviewing your source document structure(s), read about key concepts in Elasticsearch, split the data into different (Elasticsearch) indices and ideally also create a so-called mapping.

Daniel