# Indexing performance degrading over time

**URL:** <https://discuss.elastic.co/t/indexing-performance-degrading-over-time/40229>\
**Category:** Elasticsearch\
**Created:** [January 27, 2016, 12:30pm UTC](https://discuss.elastic.co/t/indexing-performance-degrading-over-time/40229 "2016-01-27T12:30:27Z")\
**Posts on this page:** 1\
**Showing post:** 26

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [February 12, 2016, 4:16pm UTC](https://discuss.elastic.co/t/indexing-performance-degrading-over-time/40229/26 "2016-02-12T16:16:50Z")

</div>

Hmm your hot threads output on that "still busy even after stopping bulk indexing" node is unhealthy, with nearly all threads doing this:

```auto
  97.7% (976.8ms out of 1s) cpu usage by thread 'elasticsearch[test_data_11-d2][bulk][T#7]'
     10/10 snapshots sharing following 22 elements
       java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(Unknown Source)
       java.lang.ThreadLocal$ThreadLocalMap.remove(Unknown Source)
       java.lang.ThreadLocal$ThreadLocalMap.access$200(Unknown Source)
       java.lang.ThreadLocal.remove(Unknown Source)
       java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(Unknown Source)
       java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(Unknown Source)
       java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(Unknown Source)
       org.elasticsearch.common.util.concurrent.ReleasableLock.close(ReleasableLock.java:49)
       org.elasticsearch.index.engine.InternalEngine.create(InternalEngine.java:365)
       org.elasticsearch.index.shard.IndexShard.create(IndexShard.java:531)
       org.elasticsearch.index.engine.Engine$Create.execute(Engine.java:810)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:476)
       org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:69)

```

It's as if you have waaay too many `ThreadLocal` instances and these threads are stuck walking through all of them pruning the now unreferenced ones.

Which Java version are you using?

Do you still have any settings increasing e.g. thread pool sizes?

---

_[View the full topic](https://discuss.elastic.co/t/indexing-performance-degrading-over-time/40229)._