Near Realtime Question


(James Cook) #1

We are using ElasticSearch as a datastore in place of a database. Kimchy
will be quick to add that this is not advisable at the moment due to its
beta status. (We also dupe data to MySQL for the present time.)

For most of our use cases, we do fine with ES NRT, but there are a few spots
when we have to issue a refresh before continuing. I was wondering if it
would be possible to add a parameter (similar to op_type) that can be passed
along with a POST or PUT to indicate that the call should not reply until
the document is indexed.

We have a multi-tier system, and this would help us by not requiring two
REST calls from the middle tier to our web services tier where ES lives.

-- jim


(Shay Banon) #2

Hi,

First, the document is always indexed, and added to a transaction log
before the operations returned. Its the visibility of it for search that is
near real time. And yes, there is a flag for that, index and delete APIs
accept a refresh parameter that will cause the relevant shard they hit to
refresh before returning a result (note, found a small problem with it in
0.12, fixed in master).

-shay.banon

On Thu, Nov 4, 2010 at 9:12 PM, James Cook jcook@tracermedia.com wrote:

We are using ElasticSearch as a datastore in place of a database. Kimchy
will be quick to add that this is not advisable at the moment due to its
beta status. (We also dupe data to MySQL for the present time.)

For most of our use cases, we do fine with ES NRT, but there are a few
spots when we have to issue a refresh before continuing. I was wondering if
it would be possible to add a parameter (similar to op_type) that can be
passed along with a POST or PUT to indicate that the call should not reply
until the document is indexed.

We have a multi-tier system, and this would help us by not requiring two
REST calls from the middle tier to our web services tier where ES lives.

-- jim


(James Cook) #3

:slight_smile:

Awesome, didn't know that parameter existed.

On Thu, Nov 4, 2010 at 3:20 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

First, the document is always indexed, and added to a transaction log
before the operations returned. Its the visibility of it for search that is
near real time. And yes, there is a flag for that, index and delete APIs
accept a refresh parameter that will cause the relevant shard they hit to
refresh before returning a result (note, found a small problem with it in
0.12, fixed in master).

-shay.banon

On Thu, Nov 4, 2010 at 9:12 PM, James Cook jcook@tracermedia.com wrote:

We are using ElasticSearch as a datastore in place of a database. Kimchy
will be quick to add that this is not advisable at the moment due to its
beta status. (We also dupe data to MySQL for the present time.)

For most of our use cases, we do fine with ES NRT, but there are a few
spots when we have to issue a refresh before continuing. I was wondering if
it would be possible to add a parameter (similar to op_type) that can be
passed along with a POST or PUT to indicate that the call should not reply
until the document is indexed.

We have a multi-tier system, and this would help us by not requiring two
REST calls from the middle tier to our web services tier where ES lives.

-- jim


(Diptamay) #4

You could alternately perform with version 0.12.1
_client.admin().indices().prepareRefresh("index").execute(); since the
refresh boolean parameter ain't being honored in 0.12.1.

On Nov 4, 3:51 pm, James Cook jc...@tracermedia.com wrote:

:slight_smile:

Awesome, didn't know that parameter existed.

On Thu, Nov 4, 2010 at 3:20 PM, Shay Banon shay.ba...@elasticsearch.comwrote:

Hi,

First, the document is always indexed, and added to a transaction log
before the operations returned. Its the visibility of it for search that is
near real time. And yes, there is a flag for that, index and delete APIs
accept a refresh parameter that will cause the relevant shard they hit to
refresh before returning a result (note, found a small problem with it in
0.12, fixed in master).

-shay.banon

On Thu, Nov 4, 2010 at 9:12 PM, James Cook jc...@tracermedia.com wrote:

We are using ElasticSearch as a datastore in place of a database. Kimchy
will be quick to add that this is not advisable at the moment due to its
beta status. (We also dupe data to MySQL for the present time.)

For most of our use cases, we do fine with ES NRT, but there are a few
spots when we have to issue a refresh before continuing. I was wondering if
it would be possible to add a parameter (similar to op_type) that can be
passed along with a POST or PUT to indicate that the call should not reply
until the document is indexed.

We have a multi-tier system, and this would help us by not requiring two
REST calls from the middle tier to our web services tier where ES lives.

-- jim


(system) #5