# Missing transactions after starting new node **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2010, 11:34pm UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612 "2010-11-30T23:34:08Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [November 30, 2010, 11:34pm UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/1 "2010-11-30T23:34:08Z")
Running 0.13.0 snapshot, local gateway. We have 78 shards, 28M docs, 170G of data. When adding a new node to the cluster, after all the shards are reallocated, I see a small difference in total number of docs (seen via REST api). During the shard re-allocations, we were actively indexing (but at a modest steady state rate). Rerunning transactions from the past hour, allowed search counts to become consistent. This happened on two different clusters. In the first there was one node and the second was added. The second time, two nodes were in the cluster and the third node was added. In both cases, all the machines in the cluster are identical hardware. Please let me know what log files you may be interested in. Thanks, David ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 1, 2010, 12:36am UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/2 "2010-12-01T00:36:32Z")
Do you remember which 0.13 snapshot you are working on? Maybe what date? I have long running tests that simulate this, and did not come across it. Can you share more info on number of indices, shards per index? I can try and simulate it here.... -shay.banon On Wed, Dec 1, 2010 at 1:34 AM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > Running 0.13.0 snapshot, local gateway. We have 78 shards, 28M docs, > 170G of data. > > When adding a new node to the cluster, after all the shards are > reallocated, I see a small difference in total number of docs (seen > via REST api). During the shard re-allocations, we were actively > indexing (but at a modest steady state rate). Rerunning transactions > from the past hour, allowed search counts to become consistent. > > This happened on two different clusters. In the first there was one > node and the second was added. The second time, two nodes were in the > cluster and the third node was added. In both cases, all the machines > in the cluster are identical hardware. > > Please let me know what log files you may be interested in. > > Thanks, > > David ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [December 1, 2010, 5:41pm UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/3 "2010-12-01T17:41:11Z")
Wow brain fart on my part, we're on 0.13.0 release. \_status dump is here [https://gist.github.com/723879](https://gist.github.com/723879) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 2, 2010, 4:50pm UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/4 "2010-12-02T16:50:28Z")
I will try and simulate it, I have a pretty extensive test that checks data does not get lost in such cases. One last question, how do you tell that the docs are different, is that based on the index status API? On Wed, Dec 1, 2010 at 7:41 PM, dbenson [dbenson@dbenson.net](mailto:dbenson@dbenson.net) wrote: > Wow brain fart on my part, we're on 0.13.0 release. > > \_status dump is here [Index Status ยท GitHub](https://gist.github.com/723879) ---
**Author:** ![dbenson](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@dbenson](https://discuss.elastic.co/u/dbenson)\ **Post date:** [December 6, 2010, 3:10pm UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/5 "2010-12-06T15:10:16Z")
Sorry for the delay in responding. The difference was visible via the REST api, when issuing a search across all indexes, all docs [http://localhost:9200/\_search?q=\*:](http://localhost:9200/_search?q=*:)\* Issuing the same search via the Java APIs didn't show a document count difference. I had an opportunity to restart the cluster during a planned maintenance window and afterwards the document count discrepancy went away from the REST api. David On Dec 2, 9:50 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > I will try and simulate it, I have a pretty extensive test that checks data > does not get lost in such cases. One last question, how do you tell that the > docs are different, is that based on the index status API? > > On Wed, Dec 1, 2010 at 7:41 PM, dbenson [dben...@dbenson.net](mailto:dben...@dbenson.net) wrote: > > > Wow brain fart on my part, we're on 0.13.0 release. > > > \_status dump is herehttps://gist.github.com/723879 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/missing-transactions-after-starting-new-node/3612/6 "2017-07-06T04:15:39Z")
Closed on July 6, 2017, 4:15am UTC