# Elasticsearch with a 90m doc index limit? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2012, 4:39pm UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832 "2012-08-23T16:39:04Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 23, 2012, 4:39pm UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/1 "2012-08-23T16:39:04Z")
So after resolving my issue importing my issues from couchdb I am seeing a new issue. After importing about 90million docs for 9hours suddenly Elasticsearch will stop important the information. Using TRACE for couchdb I see the river is still sending the data but my \_seq is not changing. We have about 200million docs in couchdb and while I can use filter to shard it out to different docs I can see a case which we have more then 90 million docs per index so I am betting there is some configuration I am missing so I am open to ideas what i need to tweak. Thanks Zuhaib -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 24, 2012, 1:26am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/2 "2012-08-24T01:26:49Z")
After starting over again I get the same result. This time I set logging to DEBUG and I get the following: [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] [search-e1] [history][9] merge [\_1ox] done, took [1m] [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] [search-e1] [history][5] merge [\_1pc] done, took [23.6s] [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] [search-e1] [history][7] merge [\_1ph] done, took [21.6s] curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' { "cluster\_name" : "elasticsearch", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 20, "active\_shards" : 20, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } And then it stops indexing as you can see from the screenshot attached. On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > So after resolving my issue importing my issues from couchdb I am seeing a > new issue. After importing about 90million docs for 9hours suddenly > Elasticsearch will stop important the information. Using TRACE for couchdb > I see the river is still sending the data but my \_seq is not changing. We > have about 200million docs in couchdb and while I can use filter to shard > it out to different docs I can see a case which we have more then 90 > million docs per index so I am betting there is some configuration I am > missing so I am open to ideas what i need to tweak. > > Thanks > Zuhaib -- ---
**Author:** ![Patrick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick/32/1437_2.png) [@Patrick](https://discuss.elastic.co/u/Patrick)\ **Post date:** [August 24, 2012, 1:34am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/3 "2012-08-24T01:34:06Z")
Hey, Could you perhaps send along any configuration you've done, as well as any information you can provide around your system, and how many nodes are in your cluster. An alternative for you would be to create an index per 'series', or 'flow' of data (so if it's monthly data, perhaps an index per month, or whatever), and then use aliases to create groupings of indices, but without more information on your system or data, it'd be hard to suggest alternatives. In the meanwhile, lets work out why it's hitting a 'document limit' ## Patrick > **[Patrick Ancillotti on about.me](https://about.me/patrick.ancillotti)** > > I live in New York. patrick eefy net On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > After starting over again I get the same result. This time I set logging to > DEBUG and I get the following: > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] [search-e1] > [history][9] merge [\_1ox] done, took [1m] > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] [search-e1] > [history][5] merge [\_1pc] done, took [23.6s] > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] [search-e1] > [history][7] merge [\_1ph] done, took [21.6s] > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > { > "cluster\_name" : "elasticsearch", > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 20, > "active\_shards" : 20, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > And then it stops indexing as you can see from the screenshot attached. > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > So after resolving my issue importing my issues from couchdb I am seeing a > > new issue. After importing about 90million docs for 9hours suddenly > > Elasticsearch will stop important the information. Using TRACE for couchdb > > I see the river is still sending the data but my \_seq is not changing. We > > have about 200million docs in couchdb and while I can use filter to shard it > > out to different docs I can see a case which we have more then 90 million > > docs per index so I am betting there is some configuration I am missing so I > > am open to ideas what i need to tweak. > > > > Thanks > > Zuhaib > > -- -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 24, 2012, 2:16am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/4 "2012-08-24T02:16:49Z")
Patrick, Sure so this is an EC2 Instance m2.4xlarge. I am using only a single node currently with no replication to load the data on and plan is to add more nodes and replication once I am in sync or near sync with couchdb. As for configuration this this the elasitcsearch.yml: ################################### Cluster ################################### cluster.name: 'elasticsearch' node.name: 'search-e1' #################################### Index #################################### index.number\_of\_shards: 10 index.number\_of\_replicas: 0 action.auto\_create\_index: true index.mapper.dynamic: true #################################### Paths #################################### path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch ################################### Memory #################################### bootstrap.mlockall: false ################################### Varia ##################################### action.disable\_delete\_all\_indices: true (note the path.data is symlinked to an EBS mount, i am changing my chef configs to point to it directly in the future). I disabled refresh\_interval as its recommended when importing a large mount of data. I am using the couchdb river and its configuration is the following: { "type" : "couchdb", "couchdb" : { "host" : "localhost", "port" : 5984, "db" : "history", "filter" : null }, "index" : { "index" : "history", "type" : "history", "bulk\_size": "400", "bulk\_timeout": "40ms" } } I did have couchdb at one point on a remote box but still had the same problem. This is just a snapshot copy of our couchdb data for testing and when we go live I will have it upodate off an read-only replication. I could add more nodes but I wonder why that would make a difference when I am throwing so much HP with this one instance. I can do some filters but currently was looking for something drop in replacement for couchdb-lucene and in small VM testing it works great, just when going to production data its falling apart. Thanks Zuhaib On Thu, Aug 23, 2012 at 6:34 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > Hey, > > Could you perhaps send along any configuration you've done, as well as > any information you can provide around your system, and how many nodes > are in your cluster. An alternative for you would be to create an > index per 'series', or 'flow' of data (so if it's monthly data, > perhaps an index per month, or whatever), and then use aliases to > create groupings of indices, but without more information on your > system or data, it'd be hard to suggest alternatives. In the > meanwhile, lets work out why it's hitting a 'document limit' > > ## Patrick > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > patrick eefy net > > On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > After starting over again I get the same result. This time I set > > logging to > > DEBUG and I get the following: > > > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] [search-e1] > > [history][9] merge [\_1ox] done, took [1m] > > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] [search-e1] > > [history][5] merge [\_1pc] done, took [23.6s] > > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] [search-e1] > > [history][7] merge [\_1ph] done, took [21.6s] > > > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > { > > "cluster\_name" : "elasticsearch", > > "status" : "green", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "number\_of\_data\_nodes" : 1, > > "active\_primary\_shards" : 20, > > "active\_shards" : 20, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 0 > > } > > > > And then it stops indexing as you can see from the screenshot attached. > > > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > > > So after resolving my issue importing my issues from couchdb I am > > > seeing a > > > new issue. After importing about 90million docs for 9hours suddenly > > > Elasticsearch will stop important the information. Using TRACE for > > > couchdb > > > I see the river is still sending the data but my \_seq is not changing. > > > We > > > have about 200million docs in couchdb and while I can use filter to > > > shard it > > > out to different docs I can see a case which we have more then 90 > > > million > > > docs per index so I am betting there is some configuration I am missing > > > so I > > > am open to ideas what i need to tweak. > > > > > > Thanks > > > Zuhaib > > > > -- > > -- -- ---
**Author:** ![Patrick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick/32/1437_2.png) [@Patrick](https://discuss.elastic.co/u/Patrick)\ **Post date:** [August 24, 2012, 3:09am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/5 "2012-08-24T03:09:03Z")
Hi Zuhaib, Great! So, I see a few places where I could suggest improvements initially that may help with your speed of indexing, as well as your overall response time currently, and may perhaps help with your number of document problem. The biggest thing that stands out is the 'kaggillion shard problem' (tm kimchy), which is essentially you're battering that server with multiple shards on the same disks, and it's no doubt having problems keeping up with the work to all indicies at the same time. In a perfect world you would keep the number of shards to the number of nodes you have in your cluster (depending on workload, and data flow/shape), but because you cannot add additional shards post creation, alot of people tend to create a many-shard approach to attempt to 'capacity plan' in future, and which then pays the shard penalty from day 1. This should be avoided if at all possible, and things like aliases, routing, and a flow that makes sense for your data preferred. Have you watched any of the videos from the last BerlinBuzzwords? Shay did a great talk that may help explain some of this far better than I can in an email. The first part would be to use the local disks as your instance storage for your Elasticsearch nodes, and use EBS as your 'gateway', or even S3. If you're just doing this for indexing, the faster you can do the indexing, the better, and you can get faster access to the local disks, and not pay a latency penalty. This would allow you to restore your cluster quickly from your gateway, and keep backups with either snaps to S3, or direct storage in S3. You can then shut your cluster down later, and restore from gateway should you so wish, but the gateway will ensure persistence. The second is that you're paying a LOT for that instance right now, and you're not going to see the same results as if you built say smaller instances, and more of them. Changing to that sort of a model should (when added to the local storage change) see a significant change in the speeds of the overall solution. Shards need to be spread out to gain from them, so why don't you give it a try on 'large' instances, or even 'extra-large' instances? Just, try keep the number higher than 1 馃槈 The third part would be the 90m document hit, I don't know of a limit myself that would keep you to 90 million documents, but that said, it seems like a bad idea, and in my experience, there is more to be gained from creating smaller indices with less shards, and using routing, aliases and filters to hit the document tree that makes sense for your workload, what you really want to keep your searches as fast as possible is to help ES by hitting as few shards / indices as possible, but of course if that isn't possible, you can throw more hardware at it. Can you gist a copy of an example document perhaps? and describe how you're importing your documents? Do you plan on using the couchdb river? ## Patrick > **[Patrick Ancillotti on about.me](https://about.me/patrick.ancillotti)** > > I live in New York. patrick eefy net On Thu, Aug 23, 2012 at 8:16 PM, Zuhaib Siddique [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > Patrick, > > Sure so this is an EC2 Instance m2.4xlarge. I am using only a single node > currently with no replication to load the data on and plan is to add more > nodes and replication once I am in sync or near sync with couchdb. As for > configuration this this the elasitcsearch.yml: > > ################################### Cluster > ################################### > > cluster.name: 'elasticsearch' > node.name: 'search-e1' > > #################################### Index > #################################### > > index.number\_of\_shards: 10 > index.number\_of\_replicas: 0 > action.auto\_create\_index: true > index.mapper.dynamic: true > > #################################### Paths > #################################### > > path.data: /var/lib/elasticsearch > path.logs: /var/log/elasticsearch > > ################################### Memory > #################################### > > bootstrap.mlockall: false > > ################################### Varia > ##################################### > > action.disable\_delete\_all\_indices: true > > (note the path.data is symlinked to an EBS mount, i am changing my chef > configs to point to it directly in the future). I disabled refresh\_interval > as its recommended when importing a large mount of data. I am using the > couchdb river and its configuration is the following: > > { > "type" : "couchdb", > "couchdb" : { > "host" : "localhost", > "port" : 5984, > "db" : "history", > "filter" : null > }, > "index" : { > "index" : "history", > "type" : "history", > "bulk\_size": "400", > "bulk\_timeout": "40ms" > } > } > > I did have couchdb at one point on a remote box but still had the same > problem. This is just a snapshot copy of our couchdb data for testing and > when we go live I will have it upodate off an read-only replication. > > I could add more nodes but I wonder why that would make a difference when I > am throwing so much HP with this one instance. I can do some filters but > currently was looking for something drop in replacement for couchdb-lucene > and in small VM testing it works great, just when going to production data > its falling apart. > > Thanks > Zuhaib > On Thu, Aug 23, 2012 at 6:34 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > > > Hey, > > > > Could you perhaps send along any configuration you've done, as well as > > any information you can provide around your system, and how many nodes > > are in your cluster. An alternative for you would be to create an > > index per 'series', or 'flow' of data (so if it's monthly data, > > perhaps an index per month, or whatever), and then use aliases to > > create groupings of indices, but without more information on your > > system or data, it'd be hard to suggest alternatives. In the > > meanwhile, lets work out why it's hitting a 'document limit' > > > > ## Patrick > > > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > > patrick eefy net > > > > On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > > > After starting over again I get the same result. This time I set > > > logging to > > > DEBUG and I get the following: > > > > > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] [search-e1] > > > [history][9] merge [\_1ox] done, took [1m] > > > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] [search-e1] > > > [history][5] merge [\_1pc] done, took [23.6s] > > > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] [search-e1] > > > [history][7] merge [\_1ph] done, took [21.6s] > > > > > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > > { > > > "cluster\_name" : "elasticsearch", > > > "status" : "green", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 1, > > > "number\_of\_data\_nodes" : 1, > > > "active\_primary\_shards" : 20, > > > "active\_shards" : 20, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 0 > > > } > > > > > > And then it stops indexing as you can see from the screenshot attached. > > > > > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > > > > > So after resolving my issue importing my issues from couchdb I am > > > > seeing a > > > > new issue. After importing about 90million docs for 9hours suddenly > > > > Elasticsearch will stop important the information. Using TRACE for > > > > couchdb > > > > I see the river is still sending the data but my \_seq is not changing. > > > > We > > > > have about 200million docs in couchdb and while I can use filter to > > > > shard it > > > > out to different docs I can see a case which we have more then 90 > > > > million > > > > docs per index so I am betting there is some configuration I am missing > > > > so I > > > > am open to ideas what i need to tweak. > > > > > > > > Thanks > > > > Zuhaib > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 24, 2012, 3:36am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/6 "2012-08-24T03:36:01Z")
Patrick, Let me see if I can get a example doc but its pretty simple as you can imagine, think what you would need for chat history searching (body, to, from etc) with some extra stuff we do. Nothing fancy like geo data and etc, the only tweaking I do to the mapping is multitype the date as I had issues sorting by date at first. A lot of what you said is what i originally planned. The design is going to be 4 elasticsearch boxes all m1.xlarges using S3. I decided to go with one large box and now remove S3 was purely as a troubleshooting method as I ran in problems with index speed pulling from couchdb and now this 90million limit it seems to hit. As for EBS i am using the high IO drives and I see them hovering around 800 to 1000 op/s. You can see from my screenshot you see the box was busy doing a lot of work and then suddenly it goes idle and the log confirms that is doing nothing. I would expect that if the system or drive was not able to keep up the system would try to process the data but hitting some sort of wall, currently its idle. Now if i put the couchdb river log in TRACE i see that its still pulling data from couchdb but its doing nothing with it, its not indexing it or even bulking it up and that is what bothers me. And as far as my understanding with couchdb-river it only runs on a single node and yes the plan was to stick with couchdb-river. I understand that breaking up the data would be best and we already have some logically partitions I can think of to implement, I just wonder what would happens if each partition grows very large, our current projects says by this time next week we should have 2 billion chat history to search over. The plan was move to this and then address the problem later but I think I need to do some more planing in to this. I will google search for those videos and give them a watch. Thanks Zuhaib On Thu, Aug 23, 2012 at 8:09 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > Hi Zuhaib, > > Great! So, I see a few places where I could suggest improvements > initially that may help with your speed of indexing, as well as your > overall response time currently, and may perhaps help with your number > of document problem. The biggest thing that stands out is the > 'kaggillion shard problem' (tm kimchy), which is essentially you're > battering that server with multiple shards on the same disks, and it's > no doubt having problems keeping up with the work to all indicies at > the same time. In a perfect world you would keep the number of shards > to the number of nodes you have in your cluster (depending on > workload, and data flow/shape), but because you cannot add additional > shards post creation, alot of people tend to create a many-shard > approach to attempt to 'capacity plan' in future, and which then pays > the shard penalty from day 1. This should be avoided if at all > possible, and things like aliases, routing, and a flow that makes > sense for your data preferred. Have you watched any of the videos from > the last BerlinBuzzwords? Shay did a great talk that may help explain > some of this far better than I can in an email. > > The first part would be to use the local disks as your instance > storage for your Elasticsearch nodes, and use EBS as your 'gateway', > or even S3. If you're just doing this for indexing, the faster you can > do the indexing, the better, and you can get faster access to the > local disks, and not pay a latency penalty. This would allow you to > restore your cluster quickly from your gateway, and keep backups with > either snaps to S3, or direct storage in S3. You can then shut your > cluster down later, and restore from gateway should you so wish, but > the gateway will ensure persistence. > > The second is that you're paying a LOT for that instance right now, > and you're not going to see the same results as if you built say > smaller instances, and more of them. Changing to that sort of a model > should (when added to the local storage change) see a significant > change in the speeds of the overall solution. Shards need to be spread > out to gain from them, so why don't you give it a try on 'large' > instances, or even 'extra-large' instances? Just, try keep the number > higher than 1 馃槈 > > The third part would be the 90m document hit, I don't know of a limit > myself that would keep you to 90 million documents, but that said, it > seems like a bad idea, and in my experience, there is more to be > gained from creating smaller indices with less shards, and using > routing, aliases and filters to hit the document tree that makes sense > for your workload, what you really want to keep your searches as fast > as possible is to help ES by hitting as few shards / indices as > possible, but of course if that isn't possible, you can throw more > hardware at it. > > Can you gist a copy of an example document perhaps? and describe how > you're importing your documents? Do you plan on using the couchdb > river? > > ## Patrick > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > patrick eefy net > > On Thu, Aug 23, 2012 at 8:16 PM, Zuhaib Siddique > [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > Patrick, > > > > Sure so this is an EC2 Instance m2.4xlarge. I am using only a single node > > currently with no replication to load the data on and plan is to add more > > nodes and replication once I am in sync or near sync with couchdb. As > > for > > configuration this this the elasitcsearch.yml: > > > > ################################### Cluster > > ################################### > > > > cluster.name: 'elasticsearch' > > node.name: 'search-e1' > > > > #################################### Index > > #################################### > > > > index.number\_of\_shards: 10 > > index.number\_of\_replicas: 0 > > action.auto\_create\_index: true > > index.mapper.dynamic: true > > > > #################################### Paths > > #################################### > > > > path.data: /var/lib/elasticsearch > > path.logs: /var/log/elasticsearch > > > > ################################### Memory > > #################################### > > > > bootstrap.mlockall: false > > > > ################################### Varia > > ##################################### > > > > action.disable\_delete\_all\_indices: true > > > > (note the path.data is symlinked to an EBS mount, i am changing my chef > > configs to point to it directly in the future). I disabled > > refresh\_interval > > as its recommended when importing a large mount of data. I am using the > > couchdb river and its configuration is the following: > > > > { > > "type" : "couchdb", > > "couchdb" : { > > "host" : "localhost", > > "port" : 5984, > > "db" : "history", > > "filter" : null > > }, > > "index" : { > > "index" : "history", > > "type" : "history", > > "bulk\_size": "400", > > "bulk\_timeout": "40ms" > > } > > } > > > > I did have couchdb at one point on a remote box but still had the same > > problem. This is just a snapshot copy of our couchdb data for testing > > and > > when we go live I will have it upodate off an read-only replication. > > > > I could add more nodes but I wonder why that would make a difference > > when I > > am throwing so much HP with this one instance. I can do some filters but > > currently was looking for something drop in replacement for > > couchdb-lucene > > and in small VM testing it works great, just when going to production > > data > > its falling apart. > > > > Thanks > > Zuhaib > > On Thu, Aug 23, 2012 at 6:34 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > > > > > Hey, > > > > > > Could you perhaps send along any configuration you've done, as well as > > > any information you can provide around your system, and how many nodes > > > are in your cluster. An alternative for you would be to create an > > > index per 'series', or 'flow' of data (so if it's monthly data, > > > perhaps an index per month, or whatever), and then use aliases to > > > create groupings of indices, but without more information on your > > > system or data, it'd be hard to suggest alternatives. In the > > > meanwhile, lets work out why it's hitting a 'document limit' > > > > > > ## Patrick > > > > > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > > > patrick eefy net > > > > > > On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) > > > wrote: > > > > > > > After starting over again I get the same result. This time I set > > > > logging to > > > > DEBUG and I get the following: > > > > > > > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] > > > > [search-e1] > > > > [history][9] merge [\_1ox] done, took [1m] > > > > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] > > > > [search-e1] > > > > [history][5] merge [\_1pc] done, took [23.6s] > > > > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] > > > > [search-e1] > > > > [history][7] merge [\_1ph] done, took [21.6s] > > > > > > > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > > > { > > > > "cluster\_name" : "elasticsearch", > > > > "status" : "green", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 1, > > > > "number\_of\_data\_nodes" : 1, > > > > "active\_primary\_shards" : 20, > > > > "active\_shards" : 20, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 0 > > > > } > > > > > > > > And then it stops indexing as you can see from the screenshot > > > > attached. > > > > > > > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > > > > > > > So after resolving my issue importing my issues from couchdb I am > > > > > seeing a > > > > > new issue. After importing about 90million docs for 9hours suddenly > > > > > Elasticsearch will stop important the information. Using TRACE for > > > > > couchdb > > > > > I see the river is still sending the data but my \_seq is not > > > > > changing. > > > > > We > > > > > have about 200million docs in couchdb and while I can use filter to > > > > > shard it > > > > > out to different docs I can see a case which we have more then 90 > > > > > million > > > > > docs per index so I am betting there is some configuration I am > > > > > missing > > > > > so I > > > > > am open to ideas what i need to tweak. > > > > > > > > > > Thanks > > > > > Zuhaib > > > > > > > > -- > > > > > > -- > > > > -- > > -- -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 24, 2012, 4:32am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/7 "2012-08-24T04:32:18Z")
I'm just wondering if your problem is relative to your document content. If ES stops each time at the same document... Or, perhaps the \_changes API gives a sequence number which is too high for ES. I don't remember so far the Java type used by the river to deal with sequence (lastseq). I suggest that you identify the lastseq value in the couchDb river metadata and report it here. Then, try to get the document content from couchDb relative to this sequence number (use changes API) and fetch also the sequence+1 document. Perhaps, there's something weird with your document (encoding problem...) HTH -- David 馃槈 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 24 ao没t 2012 脿 05:36, Zuhaib Siddique [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) a 茅crit : > Patrick, > > Let me see if I can get a example doc but its pretty simple as you can imagine, think what you would need for chat history searching (body, to, from etc) with some extra stuff we do. Nothing fancy like geo data and etc, the only tweaking I do to the mapping is multitype the date as I had issues sorting by date at first. > > A lot of what you said is what i originally planned. The design is going to be 4 elasticsearch boxes all m1.xlarges using S3. I decided to go with one large box and now remove S3 was purely as a troubleshooting method as I ran in problems with index speed pulling from couchdb and now this 90million limit it seems to hit. As for EBS i am using the high IO drives and I see them hovering around 800 to 1000 op/s. You can see from my screenshot you see the box was busy doing a lot of work and then suddenly it goes idle and the log confirms that is doing nothing. I would expect that if the system or drive was not able to keep up the system would try to process the data but hitting some sort of wall, currently its idle. Now if i put the couchdb river log in TRACE i see that its still pulling data from couchdb but its doing nothing with it, its not indexing it or even bulking it up and that is what bothers me. And as far as my understanding with couchdb-river it only runs on a single node and yes the plan was to stick with couchdb-river. > > I understand that breaking up the data would be best and we already have some logically partitions I can think of to implement, I just wonder what would happens if each partition grows very large, our current projects says by this time next week we should have 2 billion chat history to search over. The plan was move to this and then address the problem later but I think I need to do some more planing in to this. I will google search for those videos and give them a watch. > > Thanks > Zuhaib > > On Thu, Aug 23, 2012 at 8:09 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > Hi Zuhaib, > > Great! So, I see a few places where I could suggest improvements > initially that may help with your speed of indexing, as well as your > overall response time currently, and may perhaps help with your number > of document problem. The biggest thing that stands out is the > 'kaggillion shard problem' (tm kimchy), which is essentially you're > battering that server with multiple shards on the same disks, and it's > no doubt having problems keeping up with the work to all indicies at > the same time. In a perfect world you would keep the number of shards > to the number of nodes you have in your cluster (depending on > workload, and data flow/shape), but because you cannot add additional > shards post creation, alot of people tend to create a many-shard > approach to attempt to 'capacity plan' in future, and which then pays > the shard penalty from day 1. This should be avoided if at all > possible, and things like aliases, routing, and a flow that makes > sense for your data preferred. Have you watched any of the videos from > the last BerlinBuzzwords? Shay did a great talk that may help explain > some of this far better than I can in an email. > > The first part would be to use the local disks as your instance > storage for your Elasticsearch nodes, and use EBS as your 'gateway', > or even S3. If you're just doing this for indexing, the faster you can > do the indexing, the better, and you can get faster access to the > local disks, and not pay a latency penalty. This would allow you to > restore your cluster quickly from your gateway, and keep backups with > either snaps to S3, or direct storage in S3. You can then shut your > cluster down later, and restore from gateway should you so wish, but > the gateway will ensure persistence. > > The second is that you're paying a LOT for that instance right now, > and you're not going to see the same results as if you built say > smaller instances, and more of them. Changing to that sort of a model > should (when added to the local storage change) see a significant > change in the speeds of the overall solution. Shards need to be spread > out to gain from them, so why don't you give it a try on 'large' > instances, or even 'extra-large' instances? Just, try keep the number > higher than 1 馃槈 > > The third part would be the 90m document hit, I don't know of a limit > myself that would keep you to 90 million documents, but that said, it > seems like a bad idea, and in my experience, there is more to be > gained from creating smaller indices with less shards, and using > routing, aliases and filters to hit the document tree that makes sense > for your workload, what you really want to keep your searches as fast > as possible is to help ES by hitting as few shards / indices as > possible, but of course if that isn't possible, you can throw more > hardware at it. > > Can you gist a copy of an example document perhaps? and describe how > you're importing your documents? Do you plan on using the couchdb > river? > > ## Patrick > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > patrick eefy net > > On Thu, Aug 23, 2012 at 8:16 PM, Zuhaib Siddique > [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > Patrick, > > > > Sure so this is an EC2 Instance m2.4xlarge. I am using only a single node > > currently with no replication to load the data on and plan is to add more > > nodes and replication once I am in sync or near sync with couchdb. As for > > configuration this this the elasitcsearch.yml: > > > > ################################### Cluster > > ################################### > > > > cluster.name: 'elasticsearch' > > node.name: 'search-e1' > > > > #################################### Index > > #################################### > > > > index.number\_of\_shards: 10 > > index.number\_of\_replicas: 0 > > action.auto\_create\_index: true > > index.mapper.dynamic: true > > > > #################################### Paths > > #################################### > > > > path.data: /var/lib/elasticsearch > > path.logs: /var/log/elasticsearch > > > > ################################### Memory > > #################################### > > > > bootstrap.mlockall: false > > > > ################################### Varia > > ##################################### > > > > action.disable\_delete\_all\_indices: true > > > > (note the path.data is symlinked to an EBS mount, i am changing my chef > > configs to point to it directly in the future). I disabled refresh\_interval > > as its recommended when importing a large mount of data. I am using the > > couchdb river and its configuration is the following: > > > > { > > "type" : "couchdb", > > "couchdb" : { > > "host" : "localhost", > > "port" : 5984, > > "db" : "history", > > "filter" : null > > }, > > "index" : { > > "index" : "history", > > "type" : "history", > > "bulk\_size": "400", > > "bulk\_timeout": "40ms" > > } > > } > > > > I did have couchdb at one point on a remote box but still had the same > > problem. This is just a snapshot copy of our couchdb data for testing and > > when we go live I will have it upodate off an read-only replication. > > > > I could add more nodes but I wonder why that would make a difference when I > > am throwing so much HP with this one instance. I can do some filters but > > currently was looking for something drop in replacement for couchdb-lucene > > and in small VM testing it works great, just when going to production data > > its falling apart. > > > > Thanks > > Zuhaib > > On Thu, Aug 23, 2012 at 6:34 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > > > > > Hey, > > > > > > Could you perhaps send along any configuration you've done, as well as > > > any information you can provide around your system, and how many nodes > > > are in your cluster. An alternative for you would be to create an > > > index per 'series', or 'flow' of data (so if it's monthly data, > > > perhaps an index per month, or whatever), and then use aliases to > > > create groupings of indices, but without more information on your > > > system or data, it'd be hard to suggest alternatives. In the > > > meanwhile, lets work out why it's hitting a 'document limit' > > > > > > ## Patrick > > > > > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > > > patrick eefy net > > > > > > On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > > > > > After starting over again I get the same result. This time I set > > > > logging to > > > > DEBUG and I get the following: > > > > > > > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] [search-e1] > > > > [history][9] merge [\_1ox] done, took [1m] > > > > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] [search-e1] > > > > [history][5] merge [\_1pc] done, took [23.6s] > > > > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] [search-e1] > > > > [history][7] merge [\_1ph] done, took [21.6s] > > > > > > > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > > > { > > > > "cluster\_name" : "elasticsearch", > > > > "status" : "green", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 1, > > > > "number\_of\_data\_nodes" : 1, > > > > "active\_primary\_shards" : 20, > > > > "active\_shards" : 20, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 0 > > > > } > > > > > > > > And then it stops indexing as you can see from the screenshot attached. > > > > > > > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > > > > > > > So after resolving my issue importing my issues from couchdb I am > > > > > seeing a > > > > > new issue. After importing about 90million docs for 9hours suddenly > > > > > Elasticsearch will stop important the information. Using TRACE for > > > > > couchdb > > > > > I see the river is still sending the data but my \_seq is not changing. > > > > > We > > > > > have about 200million docs in couchdb and while I can use filter to > > > > > shard it > > > > > out to different docs I can see a case which we have more then 90 > > > > > million > > > > > docs per index so I am betting there is some configuration I am missing > > > > > so I > > > > > am open to ideas what i need to tweak. > > > > > > > > > > Thanks > > > > > Zuhaib > > > > > > > > -- > > > > > > -- > > > > -- > > -- > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 24, 2012, 6:21am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/8 "2012-08-24T06:21:39Z")
Hi Patrick > Great! So, I see a few places where I could suggest improvements > initially that may help with your speed of indexing, as well as your > overall response time currently, and may perhaps help with your number > of document problem. The biggest thing that stands out is the > 'kaggillion shard problem' (tm kimchy), which is essentially you're > battering that server with multiple shards on the same disks, and it's > no doubt having problems keeping up with the work to all indicies at > the same time. To be honest, I don't think 20 shards on one box could be considered a kagillion shards. Sure, it's quite a few shards, but it's still manageable. Zuhaib - one thing you don't mention is your heap size. Also, you are enabling bootstrap.mlockall, but have you also set (and checked that it is taking effect) ulimit -l unlimited. And while we're about it, ulimit -n as well? > The second is that you're paying a LOT for that instance right now, > and you're not going to see the same results as if you built say > smaller instances, and more of them. Changing to that sort of a model > should (when added to the local storage change) see a significant > change in the speeds of the overall solution. Shards need to be spread > out to gain from them, so why don't you give it a try on 'large' > instances, or even 'extra-large' instances? Just, try keep the number > higher than 1 馃槈 For initial indexing, using a single node is not a bad idea. Before going into production, I'd add more nodes. At that stage, the ready made segment files just get copied over to the new nodes, which is probably faster than doing all the indexing on multiple nodes initially. > The third part would be the 90m document hit, I don't know of a limit > myself that would keep you to 90 million documents, Yeah, there isn't a limit. Although you may be bumping into some other limit eg memory, file handles, but then I'd expect to see something in your logs. Set your logging to DEBUG and see if you get any more information. How many documents are you expecting? Are you absolutely sure that all of the IDs are unique? Something I hear quite often with couch db is that not all documents have been indexed, but then it turns out that there are ID clashes, so newer documents just overwrite older documents. clint -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 24, 2012, 6:29am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/9 "2012-08-24T06:29:48Z")
David, I was thinking the same thing, so when it stopped index I turned on TRACE for the river logging and I checked out the data it pulled first and it looks normal with normal encoding. The seq number is "seq":96967136 but its not always the same it stops at that is just the one I investigated last, currently it stopped at 96967115 Looking at the river source code it seems to be a string(?) ` private class Slurper implements Runnable { @SuppressWarnings({"unchecked"}) @Override public void run() {` ` ``` while (true) { if (closed) { return; } String lastSeq = null; try { ``` ` I would expect an error or something thrown if it it had a bad encoding or something, but, again the data is totally searchable currently using couchdb-lucene. Clinton, if you see the screenshot you will see I have a lot of heap memory 馃檪 close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit you see in that screenshot and I confirmed that. I am searching for the document Id and I dont see any conflicts on couchdb. Zuhaib On Thu, Aug 23, 2012 at 9:32 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > I'm just wondering if your problem is relative to your document content. > If ES stops each time at the same document... > > Or, perhaps the \_changes API gives a sequence number which is too high for > ES. I don't remember so far the Java type used by the river to deal with > sequence (lastseq). > > I suggest that you identify the lastseq value in the couchDb river > metadata and report it here. Then, try to get the document content from > couchDb relative to this sequence number (use changes API) and fetch also > the sequence+1 document. > > Perhaps, there's something weird with your document (encoding problem...) > > HTH > > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 24 ao没t 2012 脿 05:36, Zuhaib Siddique [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) a > 茅crit : > > Patrick, > > Let me see if I can get a example doc but its pretty simple as you > can imagine, think what you would need for chat history searching (body, > to, from etc) with some extra stuff we do. Nothing fancy like geo data and > etc, the only tweaking I do to the mapping is multitype the date as I had > issues sorting by date at first. > > A lot of what you said is what i originally planned. The design is going > to be 4 elasticsearch boxes all m1.xlarges using S3. I decided to go with > one large box and now remove S3 was purely as a troubleshooting method as I > ran in problems with index speed pulling from couchdb and now this > 90million limit it seems to hit. As for EBS i am using the high IO drives > and I see them hovering around 800 to 1000 op/s. You can see from my > screenshot you see the box was busy doing a lot of work and then suddenly > it goes idle and the log confirms that is doing nothing. I would expect > that if the system or drive was not able to keep up the system would try to > process the data but hitting some sort of wall, currently its idle. Now if > i put the couchdb river log in TRACE i see that its still pulling data from > couchdb but its doing nothing with it, its not indexing it or even bulking > it up and that is what bothers me. And as far as my understanding with > couchdb-river it only runs on a single node and yes the plan was to stick > with couchdb-river. > > I understand that breaking up the data would be best and we already have > some logically partitions I can think of to implement, I just wonder what > would happens if each partition grows very large, our current projects says > by this time next week we should have 2 billion chat history to search > over. The plan was move to this and then address the problem later but I > think I need to do some more planing in to this. I will google search for > those videos and give them a watch. > > Thanks > Zuhaib > > On Thu, Aug 23, 2012 at 8:09 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > > > Hi Zuhaib, > > > > Great! So, I see a few places where I could suggest improvements > > initially that may help with your speed of indexing, as well as your > > overall response time currently, and may perhaps help with your number > > of document problem. The biggest thing that stands out is the > > 'kaggillion shard problem' (tm kimchy), which is essentially you're > > battering that server with multiple shards on the same disks, and it's > > no doubt having problems keeping up with the work to all indicies at > > the same time. In a perfect world you would keep the number of shards > > to the number of nodes you have in your cluster (depending on > > workload, and data flow/shape), but because you cannot add additional > > shards post creation, alot of people tend to create a many-shard > > approach to attempt to 'capacity plan' in future, and which then pays > > the shard penalty from day 1. This should be avoided if at all > > possible, and things like aliases, routing, and a flow that makes > > sense for your data preferred. Have you watched any of the videos from > > the last BerlinBuzzwords? Shay did a great talk that may help explain > > some of this far better than I can in an email. > > > > The first part would be to use the local disks as your instance > > storage for your Elasticsearch nodes, and use EBS as your 'gateway', > > or even S3. If you're just doing this for indexing, the faster you can > > do the indexing, the better, and you can get faster access to the > > local disks, and not pay a latency penalty. This would allow you to > > restore your cluster quickly from your gateway, and keep backups with > > either snaps to S3, or direct storage in S3. You can then shut your > > cluster down later, and restore from gateway should you so wish, but > > the gateway will ensure persistence. > > > > The second is that you're paying a LOT for that instance right now, > > and you're not going to see the same results as if you built say > > smaller instances, and more of them. Changing to that sort of a model > > should (when added to the local storage change) see a significant > > change in the speeds of the overall solution. Shards need to be spread > > out to gain from them, so why don't you give it a try on 'large' > > instances, or even 'extra-large' instances? Just, try keep the number > > higher than 1 馃槈 > > > > The third part would be the 90m document hit, I don't know of a limit > > myself that would keep you to 90 million documents, but that said, it > > seems like a bad idea, and in my experience, there is more to be > > gained from creating smaller indices with less shards, and using > > routing, aliases and filters to hit the document tree that makes sense > > for your workload, what you really want to keep your searches as fast > > as possible is to help ES by hitting as few shards / indices as > > possible, but of course if that isn't possible, you can throw more > > hardware at it. > > > > Can you gist a copy of an example document perhaps? and describe how > > you're importing your documents? Do you plan on using the couchdb > > river? > > > > ## Patrick > > > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > > patrick eefy net > > > > On Thu, Aug 23, 2012 at 8:16 PM, Zuhaib Siddique > > [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) wrote: > > > > > Patrick, > > > > > > Sure so this is an EC2 Instance m2.4xlarge. I am using only a single > > > node > > > currently with no replication to load the data on and plan is to add > > > more > > > nodes and replication once I am in sync or near sync with couchdb. As > > > for > > > configuration this this the elasitcsearch.yml: > > > > > > ################################### Cluster > > > ################################### > > > > > > cluster.name: 'elasticsearch' > > > node.name: 'search-e1' > > > > > > #################################### Index > > > #################################### > > > > > > index.number\_of\_shards: 10 > > > index.number\_of\_replicas: 0 > > > action.auto\_create\_index: true > > > index.mapper.dynamic: true > > > > > > #################################### Paths > > > #################################### > > > > > > path.data: /var/lib/elasticsearch > > > path.logs: /var/log/elasticsearch > > > > > > ################################### Memory > > > #################################### > > > > > > bootstrap.mlockall: false > > > > > > ################################### Varia > > > ##################################### > > > > > > action.disable\_delete\_all\_indices: true > > > > > > (note the path.data is symlinked to an EBS mount, i am changing my chef > > > configs to point to it directly in the future). I disabled > > > refresh\_interval > > > as its recommended when importing a large mount of data. I am using the > > > couchdb river and its configuration is the following: > > > > > > { > > > "type" : "couchdb", > > > "couchdb" : { > > > "host" : "localhost", > > > "port" : 5984, > > > "db" : "history", > > > "filter" : null > > > }, > > > "index" : { > > > "index" : "history", > > > "type" : "history", > > > "bulk\_size": "400", > > > "bulk\_timeout": "40ms" > > > } > > > } > > > > > > I did have couchdb at one point on a remote box but still had the same > > > problem. This is just a snapshot copy of our couchdb data for testing > > > and > > > when we go live I will have it upodate off an read-only replication. > > > > > > I could add more nodes but I wonder why that would make a difference > > > when I > > > am throwing so much HP with this one instance. I can do some filters > > > but > > > currently was looking for something drop in replacement for > > > couchdb-lucene > > > and in small VM testing it works great, just when going to production > > > data > > > its falling apart. > > > > > > Thanks > > > Zuhaib > > > On Thu, Aug 23, 2012 at 6:34 PM, Patrick [patrick@eefy.net](mailto:patrick@eefy.net) wrote: > > > > > > > Hey, > > > > > > > > Could you perhaps send along any configuration you've done, as well as > > > > any information you can provide around your system, and how many nodes > > > > are in your cluster. An alternative for you would be to create an > > > > index per 'series', or 'flow' of data (so if it's monthly data, > > > > perhaps an index per month, or whatever), and then use aliases to > > > > create groupings of indices, but without more information on your > > > > system or data, it'd be hard to suggest alternatives. In the > > > > meanwhile, lets work out why it's hitting a 'document limit' > > > > > > > > ## Patrick > > > > > > > > [Patrick Ancillotti - New York | about.me](http://about.me/patrick.ancillotti) > > > > patrick eefy net > > > > > > > > On Thu, Aug 23, 2012 at 7:26 PM, zuhaib [zsiddique@atlassian.com](mailto:zsiddique@atlassian.com) > > > > wrote: > > > > > > > > > After starting over again I get the same result. This time I set > > > > > logging to > > > > > DEBUG and I get the following: > > > > > > > > > > [2012-08-24 01:14:09,490][DEBUG][index.merge.scheduler] > > > > > [search-e1] > > > > > [history][9] merge [\_1ox] done, took [1m] > > > > > [2012-08-24 01:14:11,899][DEBUG][index.merge.scheduler] > > > > > [search-e1] > > > > > [history][5] merge [\_1pc] done, took [23.6s] > > > > > [2012-08-24 01:14:34,954][DEBUG][index.merge.scheduler] > > > > > [search-e1] > > > > > [history][7] merge [\_1ph] done, took [21.6s] > > > > > > > > > > curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > > > > { > > > > > "cluster\_name" : "elasticsearch", > > > > > "status" : "green", > > > > > "timed\_out" : false, > > > > > "number\_of\_nodes" : 1, > > > > > "number\_of\_data\_nodes" : 1, > > > > > "active\_primary\_shards" : 20, > > > > > "active\_shards" : 20, > > > > > "relocating\_shards" : 0, > > > > > "initializing\_shards" : 0, > > > > > "unassigned\_shards" : 0 > > > > > } > > > > > > > > > > And then it stops indexing as you can see from the screenshot > > > > > attached. > > > > > > > > > > On Thursday, August 23, 2012 9:39:04 AM UTC-7, zuhaib wrote: > > > > > > > > > > > So after resolving my issue importing my issues from couchdb I am > > > > > > seeing a > > > > > > new issue. After importing about 90million docs for 9hours suddenly > > > > > > Elasticsearch will stop important the information. Using TRACE for > > > > > > couchdb > > > > > > I see the river is still sending the data but my \_seq is not > > > > > > changing. > > > > > > We > > > > > > have about 200million docs in couchdb and while I can use filter to > > > > > > shard it > > > > > > out to different docs I can see a case which we have more then 90 > > > > > > million > > > > > > docs per index so I am betting there is some configuration I am > > > > > > missing > > > > > > so I > > > > > > am open to ideas what i need to tweak. > > > > > > > > > > > > Thanks > > > > > > Zuhaib > > > > > > > > > > -- > > > > > > > > -- > > > > > > -- > > > > -- > > -- > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 24, 2012, 6:51am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/10 "2012-08-24T06:51:23Z")
Hi Zuhaib > I was thinking the same thing, so when it stopped index I turned on > TRACE for the river logging and I checked out the data it pulled first > and it looks normal with normal encoding. The seq number > is "seq":96967136 but its not always the same it stops at that is just > the one I investigated last, currently it stopped at 96967115 > > Looking at the river source code it seems to be a string(?) That's interesting - yes, ES wouldn't be able to handle that. > I would expect an error or something thrown if it it had a bad > encoding or something, but, again the data is totally searchable > currently using couchdb-lucene. lucene is different. it doesn't have a schema and isn't expecting JSON. > Clinton, > if you see the screenshot you will see I have a lot of heap memory 馃檪 > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > you see in that screenshot and I confirmed that. better to include that info in text - i don't bother looking at any screenshots 馃檪 but it sounds like the string doc is your problem. clint -- ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 24, 2012, 7:00am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/11 "2012-08-24T07:00:34Z")
Hi Clint, Why do you say that ES can not handle the seq number as a String? IMHO, I don't see what can bother ES river. When the river ask couchDb for new changes, it only append the lastseq (String) as is in the URL (\_changes API) -- David 馃槈 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 24 ao没t 2012 脿 08:51, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) a 茅crit : Hi Zuhaib > I was thinking the same thing, so when it stopped index I turned on > TRACE for the river logging and I checked out the data it pulled first > and it looks normal with normal encoding. The seq number > is "seq":96967136 but its not always the same it stops at that is just > the one I investigated last, currently it stopped at 96967115 > > Looking at the river source code it seems to be a string(?) That's interesting - yes, ES wouldn't be able to handle that. > I would expect an error or something thrown if it it had a bad > encoding or something, but, again the data is totally searchable > currently using couchdb-lucene. lucene is different. it doesn't have a schema and isn't expecting JSON. > Clinton, > if you see the screenshot you will see I have a lot of heap memory 馃檪 > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > you see in that screenshot and I confirmed that. better to include that info in text - i don't bother looking at any screenshots 馃檪 but it sounds like the string doc is your problem. clint -- -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 24, 2012, 6:50pm UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/12 "2012-08-24T18:50:01Z")
Yeah I dont see the seq ID being the problem and looking at the data around the point it stops indexing it seems to be valid with no funny encoding, we are UTF-8 and I assume that elasticsearch has no problem with this (During my testing I saw no problem in my VM). What I have done now is created two index's and using the couchdb filters to split them up based on public and private chats. I am seeing slower index time but I guess its expected as couchdbjs is now doing more work to filter the \_changes to each index. I will let it run the weekend and see where it goes but I think elasticsearch and couchdb are not primetime ready and I will have to shelve this project. Zuhaib On Fri, Aug 24, 2012 at 12:00 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > Hi Clint, > > Why do you say that ES can not handle the seq number as a String? > IMHO, I don't see what can bother ES river. When the river ask couchDb for > new changes, it only append the lastseq (String) as is in the URL (\_changes > API) > > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 24 ao没t 2012 脿 08:51, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) a 茅crit : > > Hi Zuhaib > > > I was thinking the same thing, so when it stopped index I turned on > > TRACE for the river logging and I checked out the data it pulled first > > and it looks normal with normal encoding. The seq number > > is "seq":96967136 but its not always the same it stops at that is just > > the one I investigated last, currently it stopped at 96967115 > > > > Looking at the river source code it seems to be a string(?) > > That's interesting - yes, ES wouldn't be able to handle that. > > > I would expect an error or something thrown if it it had a bad > > encoding or something, but, again the data is totally searchable > > currently using couchdb-lucene. > > lucene is different. it doesn't have a schema and isn't expecting JSON. > > > Clinton, > > if you see the screenshot you will see I have a lot of heap memory 馃檪 > > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > > you see in that screenshot and I confirmed that. > > better to include that info in text - i don't bother looking at any > screenshots 馃檪 > > but it sounds like the string doc is your problem. > > clint > > -- > > -- -- ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [August 25, 2012, 9:26am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/13 "2012-08-25T09:26:46Z")
Hi David > Why do you say that ES can not handle the seq number as a String? > IMHO, I don't see what can bother ES river. When the river ask couchDb > for new changes, it only append the lastseq (String) as is in the URL > (\_changes API) Sorry I misread that in my haste - I thought Zuhaib was saying that the document itself was just a string. clint > -- > David 馃槈 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 24 ao没t 2012 脿 08:51, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) a 茅crit : > > Hi Zuhaib > > > I was thinking the same thing, so when it stopped index I turned on > > TRACE for the river logging and I checked out the data it pulled first > > and it looks normal with normal encoding. The seq number > > is "seq":96967136 but its not always the same it stops at that is just > > the one I investigated last, currently it stopped at 96967115 > > > > Looking at the river source code it seems to be a string(?) > > That's interesting - yes, ES wouldn't be able to handle that. > > > I would expect an error or something thrown if it it had a bad > > encoding or something, but, again the data is totally searchable > > currently using couchdb-lucene. > > lucene is different. it doesn't have a schema and isn't expecting JSON. > > > Clinton, > > if you see the screenshot you will see I have a lot of heap memory 馃檪 > > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > > you see in that screenshot and I confirmed that. > > better to include that info in text - i don't bother looking at any > screenshots 馃檪 > > but it sounds like the string doc is your problem. > > clint > > -- -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [August 28, 2012, 6:28pm UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/14 "2012-08-28T18:28:25Z")
So this time I split the data in to two index's using couchdb filters and again at the 98millin seq count it stops indexing. Restarting elasticsearch will get it to index one or two doc and then it stops. Logging shows nothign but the river pulling data from couchdb so I know that link is working. This is the jstack so maybe someone with better Java skills can maybe take a look at this and figure out whats going on: Deadlock Detection: No deadlocks found. Thread 14584: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 14583: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 13373: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10820: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10426: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10425: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10424: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10423: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10422: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10421: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10420: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10419: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10418: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 10417: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9789: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9763: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9762: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9745: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9744: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9711: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9710: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9709: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9582: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9563: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9543: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9541: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9513: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9512: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9448: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9443: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9424: (state = IN\_NATIVE) - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte, int, int, int) @bci=0 (Interpreted frame) - java.net.SocketInputStream.read(byte, int, int) @bci=84, line=146 (Interpreted frame) - java.io.BufferedInputStream.read1(byte, int, int) @bci=39, line=273 (Interpreted frame) - java.io.BufferedInputStream.read(byte, int, int) @bci=49, line=334 (Interpreted frame) - sun.net.www.http.ChunkedInputStream.readAheadBlocking() @bci=38, line=543 (Compiled frame) - sun.net.www.http.ChunkedInputStream.readAhead(boolean) @bci=36, line=600 (Compiled frame) - sun.net.www.http.ChunkedInputStream.read(byte, int, int) @bci=80, line=687 (Interpreted frame) - java.io.FilterInputStream.read(byte, int, int) @bci=7, line=133 (Interpreted frame) - sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(byte, int, int) @bci=4, line=2582 (Interpreted frame) - sun.nio.cs.StreamDecoder.readBytes() @bci=130, line=282 (Compiled frame) - sun.nio.cs.StreamDecoder.implRead(char, int, int) @bci=112, line=324 (Compiled frame) - sun.nio.cs.StreamDecoder.read(char, int, int) @bci=180, line=176 (Interpreted frame) - java.io.InputStreamReader.read(char, int, int) @bci=7, line=184 (Interpreted frame) - java.io.BufferedReader.fill() @bci=145, line=153 (Interpreted frame) - java.io.BufferedReader.readLine(boolean) @bci=44, line=316 (Compiled frame) - java.io.BufferedReader.readLine() @bci=2, line=379 (Interpreted frame) - org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run() @bci=550, line=472 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9423: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Interpreted frame) - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=345 (Interpreted frame) - org.elasticsearch.river.couchdb.CouchdbRiver$Indexer.run() @bci=18, line=312 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9410: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9409: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, boolean, long) @bci=102, line=352 (Interpreted frame) - java.util.concurrent.SynchronousQueue.poll(long, java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9408: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Interpreted frame) - java.util.concurrent.ArrayBlockingQueue.put(java.lang.Object) @bci=39, line=280 (Interpreted frame) - org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run() @bci=678, line=484 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9405: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9369: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9366: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9271: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9270: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9269: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9268: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9267: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9265: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9264: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9263: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9080: (state = BLOCKED) Thread 9247: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - ## java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=838 (Interpreted frame) ## java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=66, line=998 (Interpreted frame) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=24, line=1304 (Interpreted frame) - java.util.concurrent.CountDownLatch.await() @bci=5, line=235 (Interpreted frame) - org.elasticsearch.bootstrap.Bootstrap$3.run() @bci=3, line=222 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9246: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=229 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9245: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=386 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9244: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9243: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9242: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9241: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9240: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9239: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9238: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9237: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9236: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9235: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9234: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9233: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9232: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9231: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9230: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=186 (Interpreted frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2043 (Interpreted frame) - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=386 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9172: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9171: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9170: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9169: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9168: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9167: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9166: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9165: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9164: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9163: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9162: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9161: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9160: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9159: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9158: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9157: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9156: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9154: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() @bci=57, line=223 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() @bci=1, line=35 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9150: (state = IN\_NATIVE) - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 (Compiled frame; information may be imprecise) - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled frame) - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled frame) - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 (Compiled frame) - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) - org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() @bci=23, line=229 (Interpreted frame) - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() @bci=55, line=102 (Interpreted frame) - org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9149: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run() @bci=60, line=135 (Interpreted frame) Thread 9148: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Compiled frame) - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2081 (Compiled frame) - java.util.concurrent.DelayQueue.take() @bci=57, line=193 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=688 (Compiled frame) - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=681 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=17, line=1103 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) Thread 9124: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run() @bci=18, line=374 (Interpreted frame) Thread 9096: (state = BLOCKED) Thread 9095: (state = BLOCKED) Thread 9094: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=133 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove() @bci=2, line=149 (Interpreted frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=177 (Interpreted frame) Thread 9093: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame) On Sat, Aug 25, 2012 at 2:26 AM, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com)wrote: > Hi David > > > Why do you say that ES can not handle the seq number as a String? > > IMHO, I don't see what can bother ES river. When the river ask couchDb > > for new changes, it only append the lastseq (String) as is in the URL > > (\_changes API) > > Sorry I misread that in my haste - I thought Zuhaib was saying that the > document itself was just a string. > > clint > > > -- > > David 馃槈 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 24 ao没t 2012 脿 08:51, Clinton Gormley [clint@traveljury.com](mailto:clint@traveljury.com) a 茅crit > > : > > > > Hi Zuhaib > > > > > I was thinking the same thing, so when it stopped index I turned on > > > TRACE for the river logging and I checked out the data it pulled first > > > and it looks normal with normal encoding. The seq number > > > is "seq":96967136 but its not always the same it stops at that is just > > > the one I investigated last, currently it stopped at 96967115 > > > > > > Looking at the river source code it seems to be a string(?) > > > > That's interesting - yes, ES wouldn't be able to handle that. > > > > > I would expect an error or something thrown if it it had a bad > > > encoding or something, but, again the data is totally searchable > > > currently using couchdb-lucene. > > > > lucene is different. it doesn't have a schema and isn't expecting JSON. > > > > > Clinton, > > > if you see the screenshot you will see I have a lot of heap memory 馃檪 > > > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > > > you see in that screenshot and I confirmed that. > > > > better to include that info in text - i don't bother looking at any > > screenshots 馃檪 > > > > but it sounds like the string doc is your problem. > > > > clint > > > > -- > > -- -- ---
**Author:** ![Juraj\_Vitko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juraj_vitko/32/2629_2.png) [@Juraj\_Vitko](https://discuss.elastic.co/u/Juraj_Vitko)\ **Post date:** [November 11, 2012, 6:37pm UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/15 "2012-11-11T18:37:15Z")
Just out of curiosity, did you guys solve this problem? On Tuesday, August 28, 2012 9:28:30 PM UTC+3, zuhaib wrote: > So this time I split the data in to two index's using couchdb filters and > again at the 98millin seq count it stops indexing. Restarting > elasticsearch will get it to index one or two doc and then it stops. > Logging shows nothign but the river pulling data from couchdb so I know > that link is working. This is the jstack so maybe someone with better Java > skills can maybe take a look at this and figure out whats going on: > > Deadlock Detection: > > No deadlocks found. > > Thread 14584: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 14583: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 13373: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10820: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10426: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10425: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10424: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10423: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10422: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10421: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10420: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10419: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10418: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 10417: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9789: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9763: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9762: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9745: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9744: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9711: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9710: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9709: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9582: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9563: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9543: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9541: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9513: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9512: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9448: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9443: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9424: (state = IN\_NATIVE) > > - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte, > int, int, int) @bci=0 (Interpreted frame) > - java.net.SocketInputStream.read(byte, int, int) @bci=84, line=146 > (Interpreted frame) > - java.io.BufferedInputStream.read1(byte, int, int) @bci=39, line=273 > (Interpreted frame) > - java.io.BufferedInputStream.read(byte, int, int) @bci=49, line=334 > (Interpreted frame) > - sun.net.www.http.ChunkedInputStream.readAheadBlocking() @bci=38, > line=543 (Compiled frame) > - sun.net.www.http.ChunkedInputStream.readAhead(boolean) @bci=36, > line=600 (Compiled frame) > - sun.net.www.http.ChunkedInputStream.read(byte, int, int) @bci=80, > line=687 (Interpreted frame) > - java.io.FilterInputStream.read(byte, int, int) @bci=7, line=133 > (Interpreted frame) > - > > sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(byte, > int, int) @bci=4, line=2582 (Interpreted frame) > > - sun.nio.cs.StreamDecoder.readBytes() @bci=130, line=282 (Compiled frame) > - sun.nio.cs.StreamDecoder.implRead(char, int, int) @bci=112, line=324 > (Compiled frame) > - sun.nio.cs.StreamDecoder.read(char, int, int) @bci=180, line=176 > (Interpreted frame) > - java.io.InputStreamReader.read(char, int, int) @bci=7, line=184 > (Interpreted frame) > - java.io.BufferedReader.fill() @bci=145, line=153 (Interpreted frame) > - java.io.BufferedReader.readLine(boolean) @bci=44, line=316 (Compiled > frame) > - java.io.BufferedReader.readLine() @bci=2, line=379 (Interpreted frame) > - org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run() @bci=550, > line=472 (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9423: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=186 (Interpreted frame) > - > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > @bci=42, line=2043 (Interpreted frame) > > - java.util.concurrent.ArrayBlockingQueue.take() @bci=20, line=345 > (Interpreted frame) > - org.elasticsearch.river.couchdb.CouchdbRiver$Indexer.run() @bci=18, > line=312 (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9410: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9409: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, boolean, long) @bci=174, line=453 (Compiled frame) > > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=352 (Interpreted frame) > > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=903 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9408: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=186 (Interpreted frame) > - > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > @bci=42, line=2043 (Interpreted frame) > > - java.util.concurrent.ArrayBlockingQueue.put(java.lang.Object) @bci=39, > line=280 (Interpreted frame) > - org.elasticsearch.river.couchdb.CouchdbRiver$Slurper.run() @bci=678, > line=484 (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9405: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9369: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9366: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9271: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9270: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9269: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9268: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9267: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.awaitMatch(org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=180, line=702 (Compiled frame) > > ## org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=615 (Compiled frame) > > org.elasticsearch.common.util.concurrent.jsr166y.LinkedTransferQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=9, line=1117 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=141, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9265: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9264: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9263: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9080: (state = BLOCKED) > > Thread 9247: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=186 (Interpreted frame) > - > > ## java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=838 (Interpreted frame) > > ## java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=66, line=998 (Interpreted frame) > > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) > @bci=24, line=1304 (Interpreted frame) > > - java.util.concurrent.CountDownLatch.await() @bci=5, line=235 > (Interpreted frame) > - org.elasticsearch.bootstrap.Bootstrap$3.run() @bci=3, line=222 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9246: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() > @bci=23, line=229 (Interpreted frame) > > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9245: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=186 (Interpreted frame) > - > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > @bci=42, line=2043 (Interpreted frame) > > - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=386 > (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 > (Interpreted frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9244: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9243: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9242: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9241: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9240: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9239: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9238: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9237: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9236: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9235: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9234: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9233: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9232: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9231: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9230: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame) > - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, > line=186 (Interpreted frame) > - > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() > @bci=42, line=2043 (Interpreted frame) > > - java.util.concurrent.LinkedBlockingQueue.take() @bci=29, line=386 > (Interpreted frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 > (Interpreted frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9172: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9171: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9170: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9169: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9168: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9167: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9166: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9165: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9164: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9163: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9162: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9161: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9160: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9159: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9158: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9157: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9156: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9154: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > ## org.elasticsearch.common.netty.channel.socket.nio.SelectorUtil.select(java.nio.channels.Selector) @bci=4, line=52 (Compiled frame) > > org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run() > @bci=57, line=223 (Compiled frame) > > - org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run() > @bci=1, line=35 (Interpreted frame) > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9150: (state = IN\_NATIVE) > > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=228 (Compiled > frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=83 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=87 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=98 (Compiled frame) > - > > org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink$Boss.run() > @bci=23, line=229 (Interpreted frame) > > - org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run() > @bci=55, line=102 (Interpreted frame) > - > > ## org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run() @bci=14, line=42 (Interpreted frame) > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=46, line=1110 (Interpreted frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9149: (state = BLOCKED) > > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) > - org.elasticsearch.indices.ttl.IndicesTTLService$PurgerThread.run() > @bci=60, line=135 (Interpreted frame) > > Thread 9148: (state = BLOCKED) > > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information > may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=226 (Compiled frame) > - > > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) > @bci=68, line=2081 (Compiled frame) > > - java.util.concurrent.DelayQueue.take() @bci=57, line=193 (Compiled > frame) > - > > ## java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=4, line=688 (Compiled frame) > > java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() > @bci=1, line=681 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1043 > (Compiled frame) > - > > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=17, line=1103 (Compiled frame) > > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 > (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) > > Thread 9124: (state = BLOCKED) > > - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) > - org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run() > @bci=18, line=374 (Interpreted frame) > > Thread 9096: (state = BLOCKED) > > Thread 9095: (state = BLOCKED) > > Thread 9094: (state = BLOCKED) > > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=133 > (Interpreted frame) > - java.lang.ref.ReferenceQueue.remove() @bci=2, line=149 (Interpreted > frame) > - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=177 > (Interpreted frame) > > Thread 9093: (state = BLOCKED) > > - java.lang.Object.wait(long) @bci=0 (Interpreted frame) > - java.lang.Object.wait() @bci=2, line=502 (Interpreted frame) > - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 > (Interpreted frame) > > On Sat, Aug 25, 2012 at 2:26 AM, Clinton Gormley \<[cl...@traveljury.com](mailto:cl...@traveljury.com)\ > > > wrote: > > > Hi David > > > > > Why do you say that ES can not handle the seq number as a String? > > > IMHO, I don't see what can bother ES river. When the river ask couchDb > > > for new changes, it only append the lastseq (String) as is in the URL > > > (\_changes API) > > > > Sorry I misread that in my haste - I thought Zuhaib was saying that the > > document itself was just a string. > > > > clint > > > > > -- > > > David 馃槈 > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > Le 24 ao没t 2012 脿 08:51, Clinton Gormley \<[cl...@traveljury.com](mailto:cl...@traveljury.com)\\> > > > a 茅crit : > > > > > > Hi Zuhaib > > > > > > > I was thinking the same thing, so when it stopped index I turned on > > > > TRACE for the river logging and I checked out the data it pulled first > > > > and it looks normal with normal encoding. The seq number > > > > is "seq":96967136 but its not always the same it stops at that is just > > > > the one I investigated last, currently it stopped at 96967115 > > > > > > > > Looking at the river source code it seems to be a string(?) > > > > > > That's interesting - yes, ES wouldn't be able to handle that. > > > > > > > I would expect an error or something thrown if it it had a bad > > > > encoding or something, but, again the data is totally searchable > > > > currently using couchdb-lucene. > > > > > > lucene is different. it doesn't have a schema and isn't expecting JSON. > > > > > > > Clinton, > > > > if you see the screenshot you will see I have a lot of heap memory 馃檪 > > > > close to 43GB of heap thanks to the m2.4xlarge instance. Also ulimit > > > > you see in that screenshot and I confirmed that. > > > > > > better to include that info in text - i don't bother looking at any > > > screenshots 馃檪 > > > > > > but it sounds like the string doc is your problem. > > > > > > clint > > > > > > -- > > > > -- -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 12, 2012, 8:01am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/16 "2012-11-12T08:01:52Z")
ES is doing well, it seems, there is something in the couchdb data going on. J枚rg On Sunday, November 11, 2012 7:37:15 PM UTC+1, Juraj Vitko wrote: > Just out of curiosity, did you guys solve this problem? -- ---
**Author:** ![zuhaib\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zuhaib_2/32/1896_2.png) [@zuhaib\_2](https://discuss.elastic.co/u/zuhaib_2)\ **Post date:** [November 13, 2012, 5:45am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/17 "2012-11-13T05:45:47Z")
Juraj, So it turned out the issue was on our side (couchdb) but the way elasticsearch/river handled was not ideal. We had committed some data as Int in couchdb that should have been stored as string as they where very large. Accuracy would FUBAR when sent as a JSON (Javascript has a limitation on the max number it can store and then precision goes to hell, you can google about it for more detail as I am no JS expert) and elasitsearch would just refuse to index the data yet the river kept on pulling the data. The lack of any error or anything tipping us off really set me back and it was not till i looked very closely at the data it would stop on did we figure out the problem. If i had more time I would try to dig deeper in the river code to see if it can catch an issue like this in the future but we went ahead and corrected our app and fixed the old data and it imported fine after that. Zuhaib On Mon, Nov 12, 2012 at 12:01 AM, J枚rg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > ES is doing well, it seems, there is something in the couchdb data going > on. > > J枚rg > > On Sunday, November 11, 2012 7:37:15 PM UTC+1, Juraj Vitko wrote: > > > Just out of curiosity, did you guys solve this problem? > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:04am UTC](https://discuss.elastic.co/t/elasticsearch-with-a-90m-doc-index-limit/8832/18 "2017-07-06T03:04:36Z")