1 second index refresh and eventual consistency between nodes

Elastic Stack Elasticsearch
1

Hi there,

Could someone clarify the index refresh behavior. I understand and see
in my app and tests that the writes are made visible to my (currently)
single node app at 1 second intervals.

My app will be a multi-node cluster with each app node also acting as
an embedded ES.

Question is: Will an update to master node A be made available to the
replicas within that 1 second refresh period, or is this only true for
the local node?

We have https://github.com/elasticsearch/elasticsearch/issues/1063
where there is talk of having blocking calls where a write will only
come back when it is visible everywhere. Is that possible or will that
only be possible in a single node architecture?

Thanks,

Rémy

2

When a document is indexed, it goes through the primary shard, and
replicated to the replica shards. The master node plays no part here.

Each shard (primary and replica) have a refresh interval, once a refresh is
executed, then the operations done against it from the last refresh are
visible for search. Note, get has "realtime" visibility.

More answers below:

On Wed, Aug 24, 2011 at 8:08 PM, Spring Ninja remy.gendron@ingeno.cawrote:

Hi there,

Could someone clarify the index refresh behavior. I understand and see
in my app and tests that the writes are made visible to my (currently)
single node app at 1 second intervals.

My app will be a multi-node cluster with each app node also acting as
an embedded ES.

Question is: Will an update to master node A be made available to the
replicas within that 1 second refresh period, or is this only true for
the local node?

No, indexing data means they get indexing in the relevant nodes / shards.
Refresh interval is a agnostic to that.

We have "block until refresh" indexing option · Issue #1063 · elastic/elasticsearch · GitHub
where there is talk of having blocking calls where a write will only
come back when it is visible everywhere. Is that possible or will that
only be possible in a single node architecture?

If its implemented, we should be able to implement it for a multi node case
as well.

Thanks,

Rémy

3

That would be really awesome and would simplify a lot of use cases.
Such as updating an edit screen and being able to navigate and
immediately update the associated list screen and see the results of
the update consistently. An indexAndBlockUntilReplicatedAndVisible
flag in the request would be great.

On Aug 24, 1:29 pm, Shay Banon kim...@gmail.com wrote:

When a document is indexed, it goes through the primary shard, and
replicated to the replica shards. The master node plays no part here.

Each shard (primary and replica) have a refresh interval, once a refresh is
executed, then the operations done against it from the last refresh are
visible for search. Note, get has "realtime" visibility.

More answers below:

On Wed, Aug 24, 2011 at 8:08 PM, Spring Ninja remy.gend...@ingeno.cawrote:

Hi there,

Could someone clarify the index refresh behavior. I understand and see
in my app and tests that the writes are made visible to my (currently)
single node app at 1 second intervals.

My app will be a multi-node cluster with each app node also acting as
an embedded ES.

Question is: Will an update to master node A be made available to the
replicas within that 1 second refresh period, or is this only true for
the local node?

No, indexing data means they get indexing in the relevant nodes / shards.
Refresh interval is a agnostic to that.

We havehttps://github.com/elasticsearch/elasticsearch/issues/1063
where there is talk of having blocking calls where a write will only
come back when it is visible everywhere. Is that possible or will that
only be possible in a single node architecture?

If its implemented, we should be able to implement it for a multi node case
as well.

Thanks,

Rémy

4

Just curious how this differs from passing refresh=true in the index
request?

5

It will not force a refresh on each request, instead, it will block the
response from returning until a refresh happened (the periodic one).

On Thu, Aug 25, 2011 at 4:37 PM, James Cook jcook@tracermedia.com wrote:

Just curious how this differs from passing refresh=true in the index
request?

6

I see. So the end result is the same, but forcing the refresh is disruptive
to performance.

We could definitely benefit from this functionality also. We use refresh
now, but our writes are minimal (at the moment) so we don't notice much
impact.