# Slowly Indexing speed **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2020, 12:48am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733 "2020-07-18T00:48:33Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 18, 2020, 12:48am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/1 "2020-07-18T00:48:33Z")
Hi, I insert bulk into elasticsearch with c++ client each bulk contains 30,000 documents each document has 40 numeric fields indexing duration is about 13 seconds the data in each bulk is targeted to one shard in one index Why is it taking so long? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 18, 2020, 2:28am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/2 "2020-07-18T02:28:30Z")
Always a good reference below, but do you see high CPU or IO? [https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html) Basic questions: How big is the cluster, what version, have enough RAM/CPU? All data to one index, with how many replicas? Nothing else going on in the cluster? What is your index refresh rate? Defaults to 1 sec but ideally set it longer like 30s. People often run bulk loads in parallel if they can to use parallel CPU, etc. I think (but do not know) a bulk insert is single threaded through the system. However, if you run many big bulk requests at once, you can run the system out of memory so you have to test (10MB seems a recommended batch size). ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 18, 2020, 6:58am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/3 "2020-07-18T06:58:58Z")
In the meantime I have 25 shards cluster’s size is 5gb Elasticsearch 6.x 0 replicas one node Have enough RAM/CPU data is inserted to one index only Nothing else going on in the cluster Index refresh interval is 30 sec No parallel bulk load My bulk size is already less than 10MB I will test the insert with another bulk size Thanks, Liron ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 18, 2020, 7:01am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/4 "2020-07-18T07:01:08Z")
How much RAM and CPU does the node have? Are you using local SSD as storage? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 18, 2020, 7:01am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/5 "2020-07-18T07:01:37Z")
Why do you have 25 shards on a one node system, as seems way too much? Why not one shard, up to 50GB? How much RAM & Java Heap? Try rebuilding index with 1 shard and try bulk again, at least as test. Also go to Version 7 when you can; not much difference, but good to be on new versions for new systems. ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 18, 2020, 9:18am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/6 "2020-07-18T09:18:45Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e26d0dc113d497d1a0844d01acfdb7c2fa4cd315.png) because I want to scale now I have 5GB of data but I need that the system will be able to handle with TBs of data ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00b434f741840553fdd71471c38ebe6fd104a39a.png) 4Cores/32GB RAM Java heap is the default, 8GB I will try it tomorrow ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/6/56f1a215fdbe8750dad556202a5e617b2755d322.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/3/934b271c4fc9440730f0d595c6ceedec20f6ba9f.png) Yes, I'm using SSD as storage 4Cores/32GB RAM ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 18, 2020, 11:08am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/7 "2020-07-18T11:08:06Z")
Monitor GC, CPU usage as well as disk I/O and iowait. See if you can spot any resource bottleneck based on this. Use Elasticsearch monitoring, top and iostat ( or other similar tools). ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 18, 2020, 11:54am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/8 "2020-07-18T11:54:34Z")
Thanks , I’ll try it tomorrow ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 19, 2020, 3:10am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/9 "2020-07-19T03:10:31Z")
Also raise your heap to 16GB, about 50% of VM RAM, up to about 32GB - this may make huge difference depending on all the other meta data, cluster state, etc. in there - it's easy to have no buffer space, etc. but with one node right now and testing, I'd use a single shard to get a sense of things, then when you add lots of nodes you can reindex & up your shard count. I have no idea how 32 shard indexes perform on one node with limited heap, but I'd guess poorly. ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 19, 2020, 4:24am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/10 "2020-07-19T04:24:26Z")
Thanks a lot!! You right, I also had the problem when I tried to retrieve millions of records Only the coordinator node need to be configured with that size? data node need to be configured with high heap size too? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 19, 2020, 7:36am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/11 "2020-07-19T07:36:47Z")
Depends a lot on what you are doing, data size, etc. Generally 50% of RAM up to 32GB heap is first guide, but beyond that depends on data size, how many indexes, shards, etc. Data nodes need RAM to manage the data, coordinators need RAM to manage results, etc. but all I can suggest is start with something medium, at least 8GB Heap on 16G VM and run monitoring to watch Heap, GCs, any circuit breakers, etc. Happy to give you free license to our [ELKman](http://www.elkman.io) tool also to help manage and tune. ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 19, 2020, 9:09am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/12 "2020-07-19T09:09:06Z")
each document has 40 fields and I need to be able to fetch 50 millions of records (for now) one index,2 nodes, I configured the index to be with 3 shards and one replica (just for now) do you think that I will be able to do that with 8GB heap (32GB of RAM) in the coordinator node?(the same heap size configuration in the second node) thanks a lot!!! ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 20, 2020, 3:41am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/13 "2020-07-20T03:41:37Z")
As others have mentioned in your other (numerous) threads, this is a very unusual use case for Elasticsearch, at most people want a few records, not millions, so there is really no way to know unless you test it - load up a cluster and query it and see is about all I can say - at least you only have numeric metric fields, which is good; also be sure you only get the fields you want, i.e. don't pull back \_all or \_source, etc. or pull aggregates, etc. For shards, you generally want total shards \<= node count, so for two nodes, that means one shard and one replica, BUT there are exceptions as don't let the shard get bigger than 50GB or so (maybe 75G for metrics only), and MAYBE the shard queries are multi-theaded, but I don't think so on a single node; not sure. Regardless, if you have a 32G node, use 16GB of heap, at least. Since you have an unusual use case, maybe even more if you have heap issues, but all you can do is load in 50-100M records and query them. Also, why are you doing this? What will you do with 50M records you get, and why not do 'that' thing inside ES, such as aggregations? Or split your feed, so whatever feeds ES can also feed your other target platform or use, or process with Spark, etc.? ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 20, 2020, 7:12am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/14 "2020-07-20T07:12:27Z")
thanks a lot ! There is not support for scatter graph in Kibana and it's important to analyze the data with that tool there is no aggregations in scatter graph I need millions of dots to be able to see a shape in the scatter graph ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1dfbb559ece07882ac53c0de23ebb14d7afe5418.png) The decision to use Elasticsearch was made following the need to manage big data Spark has the same search performance as Elasticsearch? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 20, 2020, 7:47am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/15 "2020-07-20T07:47:31Z")
The next question is I suppose, what are you doing with this graph? What is it used for? What is the meaningful information you want to extract from this visualization? ---
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 20, 2020, 8:11am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/16 "2020-07-20T08:11:25Z")
And every graph of any sizable data has to aggregate so do that aggregation in elasticsearch to get a base set of data you can pull into Grafana or even Excel, etc. The 'big' data part is storing data (though 50M you can do in MySQL, too) and doing some level of analyses on it, often with time or other buckets - at least let ES aggregate your 50M down to 50-50K using your Mass K1/K2/K3 buckets - I'm not a query expert, but would seem doable and you probably not care about how long it takes. ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 20, 2020, 8:35am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/17 "2020-07-20T08:35:45Z")
I can't tell what is it used for but I can tell that I learn a lot of thing just from looking at the shapes that are created by the graph 🙂 ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 20, 2020, 8:55am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/18 "2020-07-20T08:55:18Z")
In excel I need only to pick two columns without any aggregations and scatter graph is created after a while. the graph need to show me just the dots of the picked rows, it is a finite number. so there is no need for aggregation at all.. if you meant that I need to retrieve just two columns for getting this results, you right but I have much more use cases for that data(forgot to mention it). when picking Elasticsearch, I didn't care about how long it takes but I did care about just getting better search performance than oracle. It's true that I thought that I could build a scatter graph in Kibana but it doesn't matter ,I will find another solution for that. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 20, 2020, 9:05am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/19 "2020-07-20T09:05:40Z")
Can Excel handle a scatter graph if over 20 million data points? As you are looking for a scatter graph I guess you should be able to calculate the values related to the two axis per document. Once you have this you should be able to show document density using e.g. a heat map, which can be based on aggregations and will allow you to zoom into areas and view in greater detail. This should also perform and scale much better than extracting all the documents. ---
**Author:** ![liron\_gofberg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liron_gofberg/32/59823_2.png) [@liron\_gofberg](https://discuss.elastic.co/u/liron_gofberg)\ **Post date:** [July 20, 2020, 9:55am UTC](https://discuss.elastic.co/t/slowly-indexing-speed/241733/20 "2020-07-20T09:55:21Z")
I don't know yet I've tried it about 1 million dots..it was slow.. but I thought Kibana could handle it, the picture above is related to spark, Is Spark can handle with that? Is it possible to build graph with 20 millions dots in any platform(with high density) ? about heat map, I will try it thanks [Next page](https://discuss.elastic.co/t/slowly-indexing-speed/241733.md?page=2)