I tried to use postman to post over 10 million lines of data into app search, but it returned i/o timeout. Think it's too large to digest in postman, how can I import the data into app search from a json file(3gb)
Welcome!
As far as i can see, you can not load more than 10 mb of data at once. No more than 100 docs per batch. See Limits | Elastic App Search Documentation [7.11] | Elastic
So you need to split your input into smaller chunks.
If you don't want to do that by yourself you can look at Logstash. It has an output for this:
1 Like
ive read this thread
It looks similar to my use case, so wonder does it work on app search too
Hey pakwai122,
Elastic App Search and Site Search are two distinct products with unique APIs. Assuming you're referring to Elastic App Search in your original question, you'll have to follow the App Search specific documentation. The post you linked to is for Site Search. In addition to the limits documentation posted by @dadoonet, this is probably a good place to start: Documents API | Elastic App Search Documentation [7.11] | Elastic.
I hope that helps!
Ross
I shared this with @pakwai122 directly first to confirm that this works, so thought I would share here as well for posterity.
Here is a gist for a little node script you can use from the command line to upload documents from a json file to app search: App Search CLI · GitHub. This is not something officially supported by Elastic, just something that I cooked up.
Download it, give it executable permissions, then run it with a command like the following, replacing the path to the json file, the location of your running app search api, your engine name, and a private key:
app-search-cli upload ~/Documents/pokemon.json http://localhost:3002 test-engine private-kayik65qbd516q1brk724aaa
3 Likes
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.