# Get bulk data from elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [December 6, 2017, 9:55am UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489 "2017-12-06T09:55:49Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ajit](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@ajit](https://discuss.elastic.co/u/ajit)\ **Post date:** [December 6, 2017, 9:55am UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489/1 "2017-12-06T09:55:49Z")
I am using scroll API to fetch data from elasticsearch. I have 400000 records in elasticsearch. And it is taking 2min. to fetch from elasticsearch. How can I reduce this time from minutes to seconds. I have attached my code. Please provide suggestions. I want to fetch 400000 records within seconds. ``` RestClient rest = RestClient.builder(new HttpHost("172.21.153.176", 9200, "http")).build(); RestHighLevelClient restClient = new RestHighLevelClient(rest); final Scroll scroll = new Scroll(TimeValue.timeValueHours(1L)); SearchRequest searchRequest = new SearchRequest("entity_fact"); searchRequest.scroll(scroll); SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); WrapperQueryBuilder qb = QueryBuilders.wrapperQuery(queryELKPart1); searchSourceBuilder.size(10000); searchSourceBuilder.query(qb); searchRequest.source(searchSourceBuilder); SearchResponse searchResponse = restClient.search(searchRequest); String scrollId = searchResponse.getScrollId(); SearchHit[] searchHits = searchResponse.getHits().getHits(); for (int j = 0; j < searchHits.length; j++) { Map dataMap = searchHits[j].getSource(); innerList = new ArrayList(); String elkEntityName = String.valueOf(dataMap.get("entity_name")); String elkSector = String.valueOf(dataMap.get("sector")); String elkRegion = String.valueOf(dataMap.get("region")); String elkCountryHq = String.valueOf(dataMap.get("country_hq")); String elkProduct = String.valueOf(dataMap.get("product")); int timeId = Integer.valueOf(dataMap.get("time_id").toString()); double elkWallet = Double.valueOf(dataMap.get("wallet").toString()); double elkRevenue = Double.valueOf(dataMap.get("revenue").toString()); double elkSow = Double.valueOf(dataMap.get("sow").toString()); String elkGapTox = String.valueOf(dataMap.get("gap_tox")); String elkRank = String.valueOf(dataMap.get("rank")); double elkGap = Double.valueOf(dataMap.get("gap").toString()); int elkEntityId = Integer.valueOf(dataMap.get("entity_id").toString()); String elkRegionHq = String.valueOf(dataMap.get("region_hq")); String elkThisLevel = "L1"; } while (searchHits != null && searchHits.length > 0) { SearchScrollRequest scrollRequest = new SearchScrollRequest(scrollId); scrollRequest.scroll(scroll); searchResponse = restClient.searchScroll(scrollRequest); scrollId = searchResponse.getScrollId(); searchHits = searchResponse.getHits().getHits(); for (int j = 0; j < searchHits.length; j++) { Map dataMap = searchHits[j].getSource(); innerList = new ArrayList(); String elkEntityName = String.valueOf(dataMap.get("entity_name")); String elkSector = String.valueOf(dataMap.get("sector")); String elkRegion = String.valueOf(dataMap.get("region")); String elkCountryHq = String.valueOf(dataMap.get("country_hq")); String elkProduct = String.valueOf(dataMap.get("product")); int timeId = Integer.valueOf(dataMap.get("time_id").toString()); double elkWallet = Double.valueOf(dataMap.get("wallet").toString()); double elkRevenue = Double.valueOf(dataMap.get("revenue").toString()); double elkSow = Double.valueOf(dataMap.get("sow").toString()); String elkGapTox = String.valueOf(dataMap.get("gap_tox")); String elkRank = String.valueOf(dataMap.get("rank")); double elkGap = Double.valueOf(dataMap.get("gap").toString()); int elkEntityId = Integer.valueOf(dataMap.get("entity_id").toString()); String elkRegionHq = String.valueOf(dataMap.get("region_hq")); String elkThisLevel = "L1"; } } ClearScrollRequest clearScrollRequest = new ClearScrollRequest(); clearScrollRequest.addScrollId(scrollId); ClearScrollResponse clearScrollResponse = restClient.clearScroll(clearScrollRequest); boolean succeeded = clearScrollResponse.isSucceeded(); ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 6, 2017, 1:54pm UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489/2 "2017-12-06T13:54:39Z")
What does your data and query look like? What is the specification of your Elasticsearch cluster? ---
**Author:** ![ajit](https://avatars.discourse-cdn.com/v4/letter/a/a3d4f5/32.png) [@ajit](https://discuss.elastic.co/u/ajit)\ **Post date:** [December 6, 2017, 1:57pm UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489/3 "2017-12-06T13:57:11Z")
GET /entity\_fact/\_search { "query": { "bool": { "must": [ { "match\_all": {} }, { "bool": { "should": [ { "terms": { "product\_id": [39,1,13,6,2,9,12,14,20,24,29,33,37,38,7,8,3,4,5,10,11,15,16,17,18,19,125,21,22,23,25,26,27,28,30,31,32,34,35,36] } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "terms": { "sector\_id": [1,2,3,4,6,7] } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "terms": { "wallet\_location\_id": [1,2,3,4,5,6] } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "terms": { "time\_id": ["20121","20111"] } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "match\_phrase": { "client\_id": "262" } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "match\_phrase": { "gap\_tox\_id": "1" } } ], "minimum\_should\_match": 1 } }, { "bool": { "should": [ { "match\_phrase": { "data\_version\_id": "1" } } ], "minimum\_should\_match": 1 } }, ``` { "range": { "@timestamp": { "gte": 1483209000000, "lte": 1511334650586, "format": "epoch_millis" } } } ], "must_not": [] } ``` }, "\_source": {"includes": ["entity\_name","sector","region","country\_hq","product","time\_id","wallet","revenue","sow","gap\_tox","rank","gap","entity\_id","region\_hq"] } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 7, 2017, 12:49pm UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489/4 "2017-12-07T12:49:38Z")
What does disk I/O and iowait look like when you are running the query? What type of storage do you have? What is the specification/size of your cluster? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 4, 2018, 12:49pm UTC](https://discuss.elastic.co/t/get-bulk-data-from-elasticsearch/110489/5 "2018-01-04T12:49:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.