# ReadTimeout **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2018, 11:32pm UTC](https://discuss.elastic.co/t/readtimeout/136771 "2018-06-20T23:32:45Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 20, 2018, 11:32pm UTC](https://discuss.elastic.co/t/readtimeout/136771/1 "2018-06-20T23:32:45Z")
We are currently sending data to elastic search using a python plugin and suddenly started getting read time out errors on elastic. So i increased the read timeout to 60 seconds. The process ran for 6 hours and failed with same error again. We are writing about 300 million records to the index. The index write performance has been very slow as well. Can anyone help? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 5:37am UTC](https://discuss.elastic.co/t/readtimeout/136771/2 "2018-06-21T05:37:41Z")
What is the specification of your cluster? What is the average size of the document? Are you assigning document IDs to documents in the application or letting Elasticsearch assign them? Have you followed [these recommendations](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/tune-for-indexing-speed.html)? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 12:42pm UTC](https://discuss.elastic.co/t/readtimeout/136771/3 "2018-06-21T12:42:18Z")
I have a 8C/64GB/4.6TB SSD machine. Its just a single node. The documents are fairly large. (I will have to check exact size) The document IDs are assigned in the application. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 1:07pm UTC](https://discuss.elastic.co/t/readtimeout/136771/4 "2018-06-21T13:07:23Z")
If you are assigning document ids in the application, each indexing operation will also require a read as Elasticsearch will need to check if the document already exists. This tends to get slower the larger your shards are. Do you have monitoring in place so you can see if indexing performance is dropping with increased shard size? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 1:15pm UTC](https://discuss.elastic.co/t/readtimeout/136771/5 "2018-06-21T13:15:03Z")
Not sure what kind of monitoring is available out of the box but I dont have xpack. This is the community edition. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 2:35pm UTC](https://discuss.elastic.co/t/readtimeout/136771/6 "2018-06-21T14:35:35Z")
I would recommend adding monitoring, as that would make it easier to troubleshoot something like this and see patterns/trends. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 2:49pm UTC](https://discuss.elastic.co/t/readtimeout/136771/7 "2018-06-21T14:49:53Z")
But that requires me to add a license to xpack, Elastic doesnt make it easy! They want $6,600 per node for 3 nodes even if you have less than 3 nodes. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 2:51pm UTC](https://discuss.elastic.co/t/readtimeout/136771/8 "2018-06-21T14:51:25Z")
Monitoring is part of the free Basic license, which can be used in production. If you want to monitor it some other way I am sure there are other tools available as well. See [here](https://www.elastic.co/subscriptions) for more details. Which version of Elasticsearch are you using? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 3:05pm UTC](https://discuss.elastic.co/t/readtimeout/136771/9 "2018-06-21T15:05:17Z")
Oh! I did not know that. Let me install the monitoring piece then. Thanks for clarification, ill report back. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 3:54pm UTC](https://discuss.elastic.co/t/readtimeout/136771/10 "2018-06-21T15:54:42Z")
Ok I have monitoring installed via XPACK. What should I check for here? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 3:57pm UTC](https://discuss.elastic.co/t/readtimeout/136771/11 "2018-06-21T15:57:30Z")
It would be interesting to see if the indexing rate drops with the size of the shards, but that will require data to be gathered for some time. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 4:11pm UTC](https://discuss.elastic.co/t/readtimeout/136771/12 "2018-06-21T16:11:19Z")
Do you want me to send the data and watch the shards and performance in general? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 21, 2018, 4:13pm UTC](https://discuss.elastic.co/t/readtimeout/136771/13 "2018-06-21T16:13:04Z")
Watch the indexing rate and data volume as data is being indexed, then share a screenshot if you see something interesting. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 5:49pm UTC](https://discuss.elastic.co/t/readtimeout/136771/14 "2018-06-21T17:49:09Z")
I am populating the index with a small set (48MM records) right now. One thing I did notice is that the memory is only 1GB! and I changed the heap to 16GB so I should hopefully see some performance increase the next time I run something, however I am currently indexing so cant restart the services for the new memory to take into effect just yet. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 6:01pm UTC](https://discuss.elastic.co/t/readtimeout/136771/15 "2018-06-21T18:01:20Z")
During the import Elastic plugin went red. Log file tells it went out of memory since there is a Java out of memory exception. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 9:45pm UTC](https://discuss.elastic.co/t/readtimeout/136771/16 "2018-06-21T21:45:33Z")
I tried increasing the memory and I am still seeing the same issue! What else can I try? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 21, 2018, 10:29pm UTC](https://discuss.elastic.co/t/readtimeout/136771/17 "2018-06-21T22:29:33Z")
After I increased the memory the CPU usage goes to 100% and then elastic index goes red and the CPU stays 100%. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 22, 2018, 4:29am UTC](https://discuss.elastic.co/t/readtimeout/136771/18 "2018-06-22T04:29:06Z")
How much heap did you configure? Is there anything in the logs? What does the [hot threads API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-nodes-hot-threads.html) show when the node is busy? What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-stats.html)? Which version of Elasticsearch are you using? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 22, 2018, 11:31am UTC](https://discuss.elastic.co/t/readtimeout/136771/19 "2018-06-22T11:31:43Z")
I gave 16GB memory to the heap. I did not check hot threads but I can as well as the cluster stats API. After increasing the heap to 16G I see that the data moves to elastic much faster. The whole process still takes 1 hour or so and then fails. When I to a top on the linux box, elastic is having a high cpu usage. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [June 22, 2018, 7:19pm UTC](https://discuss.elastic.co/t/readtimeout/136771/20 "2018-06-22T19:19:43Z")
Actually after making the change from 1GB to 16GB I see it going down more than before. How do I get to the bottom of this? I am sure its the JVM process or something because when it goes down, I see the CPU usage to \> 100% on elastic [Next page](https://discuss.elastic.co/t/readtimeout/136771.md?page=2)