Are there limits on no. of documents ingested per request through curl?

Elastic Stack Elasticsearch
1

I have 6374 json documents that I would like to ingest.

  1. I used the curl command as follows:
    curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/projects/_bulk?pretty' --data-binary @projects.json

I found that only 1014 documents got ingested as shown by
POST /projects/_count

  1. Then I thought it could be some issue with the data, and tried to ingest
    half of the data (3200 documents)

This time I found that only around 465 documents were ingested.

  1. I tried again to narrow down the problem by ingesting the first 500 documents.

This time I found only 74 documents were ingested.

An interesting pattern I find is that only 15% of the total amount that I am trying to ingest gets into elasticsearch.

Makes me wonder if there are any thresholds or upper limits that I am exceeding in curl or elasticsearch.

Note that I tried both with and without pretty option. Same results.

I would appreciate any inputs. Thanks!

2

What does the bulk response contain? Does it indicate any errors?

1 Like
3

Apparently there were many errors in the formatting of the input data, and they prevented the documents from being ingested into elasticsearch. Had to do some tweaks to make the errors go away, and now all the documents are in elasticsearch!

The magic number of 15% of the total data everytime was what put me off the track from identifying the real issue.

Thank you for pointing out about the errors, @Christian_Dahlqvist! Much appreciate it!

Regards.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.