### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [August 31, 2011, 6:25pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/6 "2011-08-31T18:25:47Z")
What do You think about this Shay? Is it a bug, or I'm doing something
wrong?
---
### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [September 22, 2011, 11:56am UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/7 "2011-09-22T11:56:21Z")
Hello Shay. I improved my test case. This time it's in Java:
>
Problem is that in ES 0.17.\* (I checked versions 0.17.5 and 0.17.7) this
code throws an exception "Record wasn't found after 10s but can be get by
id".
But this code works well in ES 0.16.2.
It's because in ES 0.16.2 putMapping returns when mapping is successfully
created and in ES 0.17.\* it returns earlier.
If mapping isn't created before indexing, then indexing is wrong and object
can't be found by search operation.
If "sleep after put mapping" is uncommented, then this test case works
better in ES 0.17.\*.
But then there are sometimes errors: Record found after 100 ms. It means
that mapping was succesfully created before indexing of object, but object
isn't searchable yet even after refresh. Uncommenting "sleep after refresh"
solves this problem. Maybe it's because refresh returns before refreshing
operation finishes?
_I'm sure that it's a regression and execute().actionGet() should return
after operation is succesfully finished. What do you think about it?_
\*
\*
\*
\*
\*
\*
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 22, 2011, 1:01pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/8 "2011-09-22T13:01:49Z")
Hi,
Thanks for the recreation, opened an issue:
[Put mapping on a single node with new mapping will not wait for the mapping to be applied · Issue #1355 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1355). The problem is
that put mapping on a single node will not wait for the mapping to be
applied so a subsequent index request might sneak in.
Btw, if you want to fully protect against that, even in the distributed /
multi client scenario, you can simply have the mappings set as part of hte
create index API.
-shay.banon
2011/9/22 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)
> Hello Shay. I improved my test case. This time it's in Java:
> [1234552’s gists · GitHub](https://gist.github.com/1234552)
>
> Problem is that in ES 0.17.\* (I checked versions 0.17.5 and 0.17.7) this
> code throws an exception "Record wasn't found after 10s but can be get by
> id".
> But this code works well in ES 0.16.2.
> It's because in ES 0.16.2 putMapping returns when mapping is successfully
> created and in ES 0.17.\* it returns earlier.
> If mapping isn't created before indexing, then indexing is wrong and object
> can't be found by search operation.
> If "sleep after put mapping" is uncommented, then this test case works
> better in ES 0.17.\*.
> But then there are sometimes errors: Record found after 100 ms. It means
> that mapping was succesfully created before indexing of object, but object
> isn't searchable yet even after refresh. Uncommenting "sleep after refresh"
> solves this problem. Maybe it's because refresh returns before refreshing
> operation finishes?
>
> _I'm sure that it's a regression and execute().actionGet() should return
> after operation is succesfully finished. What do you think about it?_
> \*
> \*
> \*
> \*
> \*
> \*
---
### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [September 22, 2011, 1:14pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/9 "2011-09-22T13:14:15Z")
Thank you for your answer.
I modified this test case to create two nodes cluster.
And now: "Record wasn't found after 10s but can be get by id" error happens
much less but still exists.
"Record found after 100 m" error happens very often - refresh operation
doesn't work well even in two nodes cluster.
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 22, 2011, 1:22pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/10 "2011-09-22T13:22:06Z")
Can you try with the change I just committed, see if it fixes it? I ran your
test and have not encountered it. Its committed on both 0.17 branch and
master.
2011/9/22 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)
> Thank you for your answer.
> I modified this test case to create two nodes cluster.
> And now: "Record wasn't found after 10s but can be get by id" error happens
> much less but still exists.
> "Record found after 100 m" error happens very often - refresh operation
> doesn't work well even in two nodes cluster.
---
### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [September 22, 2011, 3:32pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/11 "2011-09-22T15:32:31Z")
I tried current master with your change.
PutMapping works as a charm - mapping is now always available when document
is indexed.
But there is still problem with refresh.
After indexing and refreshing sometimes object isn't found (but becomes
searchable after a while).
My test case fails now with "Record found after 100 ms" error always after
5-200 loops.
---
### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [September 23, 2011, 1:29pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/12 "2011-09-23T13:29:38Z")
I did some more tests using master revision and:
scenario 1:
create index
putMapping
sleep
loop {
index
refresh
search
}
works without problems. If index is created much before loop, then
refreshing works and document is always searchable after refresh.
scenario2:
loop {
create index
putMapping
index
refresh
search
}
fails very quickly, usually at 2nd loop. Mapping is created well but indexed
document isn't searchable after refresh if index was created recently. It
becomes searchable after a while.
scenario3:
loop {
create index
putMapping
wait for yellow status
index
refresh
search
}
doesn't fail as quickly as scenario2, but also fails. It means that even
waiting for yellow status doesn't assure as that document will be searchable
after indexing and refreshing. It becomes searchable after a while.
I'd like to be sure that I'll find all documents which was indexed before
search. How to do this?
---
### Author:  [@Frederic](https://discuss.elastic.co/u/Frederic)
#### Post date: [September 26, 2011, 12:50pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/13 "2011-09-26T12:50:52Z")
Hi there. For what it worths, same issue here but with default index
and mapping creation at indexing time. Searching items right after
indexing them and refreshing the index, does found documents but not
all of them. Sleeping 5secs after populating ES does the work. (main
parts of the test at [Populating and immediately searching ES Test · GitHub](https://gist.github.com/1242136))
On 23 sep, 10:29, Wojciech Durczyński
[wojciech.durczyn...@comarch.com](mailto:wojciech.durczyn...@comarch.com) wrote:
> I did some more tests using master revision and:
>
> scenario 1:
> create index
> putMapping
> sleep
> loop {
> index
> refresh
> search}
>
> works without problems. If index is created much before loop, then
> refreshing works and document is always searchable after refresh.
>
> scenario2:
> loop {
> create index
> putMapping
> index
> refresh
> search}
>
> fails very quickly, usually at 2nd loop. Mapping is created well but indexed
> document isn't searchable after refresh if index was created recently. It
> becomes searchable after a while.
>
> scenario3:
> loop {
> create index
> putMapping
> wait for yellow status
> index
> refresh
> search}
>
> doesn't fail as quickly as scenario2, but also fails. It means that even
> waiting for yellow status doesn't assure as that document will be searchable
> after indexing and refreshing. It becomes searchable after a while.
>
> I'd like to be sure that I'll find all documents which was indexed before
> search. How to do this?
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 27, 2011, 8:59pm UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/14 "2011-09-27T20:59:16Z")
Hey,
Ran the test again, but now with deleting the index so I can run it over
night, and it managed to fail (~3000 iteration). I guess the difference
comes from the different HW we use as its a concurrency problem. I pushed
two fixes to master that fix the ones that I managed to recreate (
[Fetch phase when searching might fail when mapping are updated with type missing · Issue #1369 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1369), and
[When refreshing, also execute the refresh operation on initializing shards to make sure we don't miss it · Issue #1370 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1370)). They mainly
relate to replica shard recovery and applying refresh / mapping to it. Can
you give it a go?
-shay.banon
2011/9/23 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)
> I did some more tests using master revision and:
>
> scenario 1:
> create index
> putMapping
> sleep
> loop {
> index
> refresh
> search
> }
> works without problems. If index is created much before loop, then
> refreshing works and document is always searchable after refresh.
>
> scenario2:
> loop {
> create index
> putMapping
> index
> refresh
> search
> }
> fails very quickly, usually at 2nd loop. Mapping is created well but
> indexed document isn't searchable after refresh if index was created
> recently. It becomes searchable after a while.
>
> scenario3:
> loop {
> create index
> putMapping
> wait for yellow status
> index
> refresh
> search
> }
> doesn't fail as quickly as scenario2, but also fails. It means that even
> waiting for yellow status doesn't assure as that document will be searchable
> after indexing and refreshing. It becomes searchable after a while.
>
> I'd like to be sure that I'll find all documents which was indexed before
> search. How to do this?
---
### Author:  [@Wojciech\_Durczynski](https://discuss.elastic.co/u/Wojciech_Durczynski)
#### Post date: [September 28, 2011, 9:28am UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/15 "2011-09-28T09:28:58Z")
I tried current master and my tests passed. Your patch seems to be working.
Is it possible to include changes from this thread (issue 1355, 1369 and
1370) in ES 0.17.8?
When do you plan to release it, and when 0.18.0?
Regards
Wojciech Durczyński
---
### Author:  [@kimchy](https://discuss.elastic.co/u/kimchy)
#### Post date: [September 28, 2011, 10:06am UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/16 "2011-09-28T10:06:29Z")
The fixes are in 0.17 branch. No concrete date for 0.18, 0.17.8 can probably
be released in a week or so.
2011/9/28 Wojciech Durczyński [wojciech.durczynski@comarch.com](mailto:wojciech.durczynski@comarch.com)
> I tried current master and my tests passed. Your patch seems to be working.
> Is it possible to include changes from this thread (issue 1355, 1369 and
> 1370) in ES 0.17.8?
> When do you plan to release it, and when 0.18.0?
>
> Regards
> Wojciech Durczyński
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:53am UTC](https://discuss.elastic.co/t/documents-missing-after-indexing-and-refreshing/5259/17 "2017-07-06T03:53:25Z")