How to fetch large data in parallel from elastic search? Using Scroll API I am able to fetch complete data from elasticsearch. But it is too slow. Takes around 40 seconds to fetch 500000 records. I am trying to use multiprocessing module in python to fetch data from scroll API simultaneously... H…

@N220 You need to use sliced scroll. You can process multiple slices simultaneously. https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#sliced-scroll

Thanks Vinayak! ... will check it out

Hi Vinayak , I am planning to implement it this way..: There are 5 shards so I will create id's 0-4 and get the 4 scroll ID's and then using multiprocessing fetch the data. Is that how it needs to be implemented?

But I am still facing the same issue: { "timed_out": false, "took": 1, "terminated_early": true, "hits": { "total": 105216, "hits": [], "max_score": null }, "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAIJyxFjRtd0o3WUc3UXJHUy1PNUdvQ2lyencAAAAAACCcsBY0bXd…

Hi Vinayak, I have resolved the issue. Now I am able to execute the queries in parallel. But I don't see any improvement in performance. Here is my code: max_slice =5 #I have created 5 process and there 5 scroll_id . process=[] for i in range(max_slice): p1=multiproces…

Are you seeing exception? Are you getting hits in the first iteration or subsequent iterations while scrolling a single slice? What version of ES are you using. early termination is not a failure. It just means whole query was not executed? if possible post query_body.

Have you tried to identify whether there is any resource limiting performance? What is the hardware specification of your cluster? What does disk I/O and iowait look like on the data nodes while you are extracting? Do you see any evidence in the logs of long or frequent GC?

Hi Vinayak, There are no exceptions. I am getting hits. Here are the details: I think the records are being fetched but not able to understand why there is no improvement in performance { "query": { "bool": { "must": [ { "match_phrase":…

Hi Christian_Dahlqvist, I don't know how to check : 1.hardware specification of your cluster Can you give me more info on the commands that I have to use..... Thanks ! ```````````````````````````

Simultaneously executing multiple queries on Scroll API to fetch Large Data

Elastic Stack Elasticsearch

N220 (srivaonf) July 10, 2020, 11:25am 15

Hi Christian_Dahlqvist,
Thanks for the response. I will surely look into it. True I did experiment with increase number of processes but there was no improvement. Any idea where I can know more about those commands?

Topic		Replies	Views
How to create parallel cursors on my dataset in Elastic search Elasticsearch	1	744	August 15, 2018
Multiple scrolls simultanious Elasticsearch	2	382	July 6, 2017
How to use elasticsearch sliced scroll with multiprocessing in python? Elasticsearch language-clients	2	3008	September 30, 2021
How to get data more than 10000 in elasticsearch Elasticsearch	27	21729	January 17, 2018
ElasticSearch for ETL - question about from-to requests and Scroll API Elasticsearch	4	1044	September 26, 2019

Simultaneously executing multiple queries on Scroll API to fetch Large Data

Related topics