# Blocked threads on read index calls **URL:** \ **Category:** Elasticsearch\ **Created:** [December 31, 2021, 4:58am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241 "2021-12-31T04:58:21Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Madhu\_Yadav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhu_yadav/32/44127_2.png) [@Madhu\_Yadav](https://discuss.elastic.co/u/Madhu_Yadav)\ **Post date:** [December 31, 2021, 4:58am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/1 "2021-12-31T04:58:21Z")
Hi, A portal built using Liferay is using Elasticsearch as a search engine with below config: Elasticsearch 7.16.2 Single Node, Around 250MB sized indices Windows Server 2019 64Bit 8 vcpu, 32GB RAM, SSD Disk 100GB OpenJDK 8 , 16GB Min and Max heap Need inputs for below: Index search requests are taking around 120 secs, with threads blocked on below code for more than 60 secs. Thread details attached and logs at [gist:0e134a7cc9a38dd4a21b7ee4ea1e8fd2 · GitHub](https://gist.github.com/madhu-liferay/0e134a7cc9a38dd4a21b7ee4ea1e8fd2) ![Screenshot from 2021-12-31 10-30-41](https://us1.discourse-cdn.com/elastic/original/3X/0/f/0f67688264ff40361f0a4fd0694876054d122aa7.png) ![Screenshot from 2021-12-31 10-30-53](https://us1.discourse-cdn.com/elastic/original/3X/4/d/4d99e079f9dfe98384b059ac01795c78d4880c2e.png) : org.apache.lucene.util.compress.LZ4.decompress(LZ4.java:112) org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:56) org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:315) org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:159) sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:717) POST /liferay-20097/\_search?typed\_keys=true&max\_concurrent\_shard\_requests=5&ignore\_unavailable=false&expand\_wildcards=open&allow\_no\_indices=true&ignore\_throttled=true&search\_type=query\_then\_fetch&batched\_reduce\_size=512&ccs\_minimize\_roundtrips=true We tried disabling the anti virus on data folder, didn't help Regards, Madhu ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 31, 2021, 10:58am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/2 "2021-12-31T10:58:55Z")
What JDK exactly are you using? None of the stock ones seem to have any way to be `BLOCKED` at `sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:717)`: ```nohighlight $ for tag in $(git tag | grep jdk8 | sort); do git show $tag:./jdk/src/share/classes/sun/nio/ch/FileChannelImpl.java | sed -ne '717p'; done | sort | uniq private static final NativeDispatcher nd = new FileDispatcherImpl(); } ``` Could you share a complete thread dump from the time of the blockage, captured using `jstack`? The screenshots and summary you've shared don't have enough detail to be useful. ---
**Author:** ![Madhu\_Yadav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhu_yadav/32/44127_2.png) [@Madhu\_Yadav](https://discuss.elastic.co/u/Madhu_Yadav)\ **Post date:** [December 31, 2021, 12:40pm UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/3 "2021-12-31T12:40:58Z")
It's Azul Zulu OpenJDK 8, will take thread dumps and share. Thanks for your response ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 31, 2021, 1:19pm UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/4 "2021-12-31T13:19:05Z")
> [@Madhu\_Yadav](#): > > It's Azul Zulu OpenJDK 8 That doesn't narrow it down much - what version exactly is it? ---
**Author:** ![Madhu\_Yadav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhu_yadav/32/44127_2.png) [@Madhu\_Yadav](https://discuss.elastic.co/u/Madhu_Yadav)\ **Post date:** [January 3, 2022, 10:47am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/5 "2022-01-03T10:47:36Z")
Hi David, Its JDK build 1.8.0\_302-b08 The corresponding Azul version is 8.56.0.21-CA-win64. Also [these](https://drive.google.com/file/d/159Kb4UiK9EOdMMmoz9BW3gpCXEr4ZM_c/view?usp=sharing) the thread dumps when the issue occurred today . /03012022/Before Restart/ are the ones when the issue occurred, post which we restarted ES and took thread dumps again for reference /03012022/After Restart/ Regards, Madhu ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 3, 2022, 3:03pm UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/6 "2022-01-03T15:03:27Z")
Your threads are all busy in the fetch phase, and the JDK apparently serialises reads to the same file on Windows: > > This means if one thread is slow reading then all the other threads will have to wait, and indeed your dumps all capture a thread `at sun.nio.ch.FileDispatcherImpl.pread0(Native Method)` which means it's waiting for the OS to respond to a read request. You will need to investigate the behaviour of that thread further to determine if it's completely stuck on a single request or if it's actually making progress just very slowly. In any case if the OS responds slowly to reads then Elasticsearch won't perform very well. I haven't found the source that corresponds with your chosen JDK but it's worth noting that the [recommended JDK is the bundled one](https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html#jvm-version) since that's the one that gets all the testing and it's much easier to answer this kind of question when using the recommended setup. I don't think it will make much difference in this case but it would still be worth ruling out that the problem is specific to your unusual JDK choice. ---
**Author:** ![Anurag\_Mittal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anurag_mittal/32/99767_2.png) [@Anurag\_Mittal](https://discuss.elastic.co/u/Anurag_Mittal)\ **Post date:** [January 3, 2022, 4:00pm UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/7 "2022-01-03T16:00:21Z")
Hi @DavidTurner Thanks for your response. This is Anurag (Madhu's colleague) I just had a quick question. Do you think changing the OS from Windows to Linux can help? If so, which version of Linux would you recommend? Thanks again for your quick responses and sharing your input. Regards, Anurag ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 3, 2022, 4:08pm UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/8 "2022-01-03T16:08:41Z")
> [@Anurag\_Mittal](#): > > Do you think changing the OS from Windows to Linux can help? If so, which version of Linux would you recommend? Mmmaybe. I mean it shouldn't be necessary, Windows is fully supported, but Linux is definitely much more common and I personally know a lot more about debugging performance issues there than on Windows. The [support matrix](https://www.elastic.co/support/matrix#matrix_os) shows all the different supported flavours, it's up to you to choose one that you're comfortable administering. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 4, 2022, 9:22am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/9 "2022-01-04T09:22:41Z")
I have opened [Improve concurrency of reads on Windows · Issue #82184 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/82184) to get thoughts from the rest of the team on this Windows-specific behaviour. ---
**Author:** ![Madhu\_Yadav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/madhu_yadav/32/44127_2.png) [@Madhu\_Yadav](https://discuss.elastic.co/u/Madhu_Yadav)\ **Post date:** [January 6, 2022, 7:33am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/10 "2022-01-06T07:33:00Z")
Changing from external JDK to bundled one didn't help, getting the same blocked threads again ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 6, 2022, 8:56am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/11 "2022-01-06T08:56:25Z")
Repeating my earlier message with my suggestion for your next steps: > [@DavidTurner](#): > > You will need to investigate the behaviour of that thread further to determine if it's completely stuck on a single request or if it's actually making progress just very slowly. In any case if the OS responds slowly to reads then Elasticsearch won't perform very well. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 3, 2022, 8:57am UTC](https://discuss.elastic.co/t/blocked-threads-on-read-index-calls/293241/12 "2022-02-03T08:57:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.