# What's best production setup for handling 1 billion records? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 29, 2016, 1:00pm UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000 "2016-02-29T13:00:31Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [February 29, 2016, 1:00pm UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/1 "2016-02-29T13:00:31Z")
Hi , I want to load 1 billion documents in elasticsearch. I am using ES **1.7.1**. Format of each document is **{** **"pid":1234,** **"sid":123,** **"dname":"[abc.com](http://abc.com)"** **}** Is it fine to load all the data into single ES node with all default settings? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 29, 2016, 10:31pm UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/2 "2016-02-29T22:31:31Z")
Depends on a few things, you really should try to see if it fits on a node of whatever size you have. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [March 1, 2016, 5:45am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/3 "2016-03-01T05:45:55Z")
I am running ES with **ES\_HEAP\_SIZE=16g** and diskspace is **500G**. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2016, 5:53am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/4 "2016-03-01T05:53:41Z")
Have you tried loading the data into a node with those specs? What was the indexing and query response like? ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [March 1, 2016, 9:42am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/5 "2016-03-01T09:42:34Z")
I have made one change in specs, I have set **index.number\_of\_shards: 1** and **index.number\_of\_replicas: 0**. Query is taking **~15 sec** but CPU utilization is around **400%.** ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 1, 2016, 9:43am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/6 "2016-03-01T09:43:27Z")
What sort of query is it? ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [March 1, 2016, 9:58am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/7 "2016-03-01T09:58:20Z")
I am using **spring-data-elasticsearch 1.3.0.RELEASE** Most of the queries are @Query("{"bool" : {"should" : [{"query\_string" : { "query" : "?0", "fields":["dname"]}} , {"query\_string" : { "query" : "?1", "fields":["dname"]}} ]}}"") for eg. return all documents where dname="[abc.com](http://abc.com)" or dname="[abc.com](http://abc.com)\*" Also logical and on pId AND sid combination. {"bool":{"must":[{"query\_string":{"query":"123","fields":["pid"],"default\_operator":"and"}},{"query\_string":{"query":"456","fields":["sid"],"default\_operator":"and"}}]}} for eg. return all documents where pid=X AND sid=Y ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [June 14, 2016, 5:00am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/8 "2016-06-14T05:00:00Z")
Can anyone help me on this ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 14, 2016, 6:10am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/9 "2016-06-14T06:10:15Z")
A leading wildcard query like that will always be slow, it's essentially the ES version of a table scan. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [June 14, 2016, 7:47am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/10 "2016-06-14T07:47:52Z")
Hi @warkolm, My query is not start with leading wildcard, It's trailing wildcard query. Thanks, Sohil ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 14, 2016, 7:50am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/11 "2016-06-14T07:50:10Z")
> [@sohilelasticsearch](#): > > "query" : "?0" > [@sohilelasticsearch](#): > > "query" : "?1" The `?` is leading though. As per [Query String Query | Elasticsearch Guide [2.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/query-dsl-query-string-query.html#_wildcards) it'll only look for a single char in there, but it still needs to scan a lot of docs to return just those. ---
**Author:** ![sohilelasticsearch](https://avatars.discourse-cdn.com/v4/letter/s/ecc23a/32.png) [@sohilelasticsearch](https://discuss.elastic.co/u/sohilelasticsearch)\ **Post date:** [June 14, 2016, 7:57am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/12 "2016-06-14T07:57:16Z")
I am using spring-data-elasticsearch. below is the code snippet. @Query("{"bool" : {"should" : [{"query\_string" : { "query" : "?0", "fields":["dname"]}} , {"query\_string" : { "query" : "?1", "fields":["dname"]}} ]}}"") Page findByDomainOrDomainStartsWith(String domain , String domain1 , Pageable pageable); "?0" -\> domain "?1" -\> domain1 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 14, 2016, 8:09am UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/13 "2016-06-14T08:09:53Z")
Ah ok, my misunderstanding then. Sorry! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:44pm UTC](https://discuss.elastic.co/t/whats-best-production-setup-for-handling-1-billion-records/43000/14 "2017-07-05T22:44:03Z")