Real time update to meta information of a document

Elastic Stack Elasticsearch
1

Hi all,

I'm researching the possibility to use ES as our primary search engine
and I'm running into a couple of issues. In my scenario the documents
look something like this (for example):
{
id: 1
some_text_fields: "some text"
some_meta_numbers : [ 1,2,3,]
}

When a new document is added to the index it is OK it will only be
visible through search after a minute. However, sometimes the numbers
in the some_meta_numbers change and I need this change to be
immediately visible. In other words, if a search is done after these
numbers have been modified, it should immediately reflect the change.
For example for all documents have 1 as number should not return a
document whose number 1 has been remove a couple of milliseconds
before.

Is this possible?

Thanks for your help,
Boaz

2

It is possible, you can add a refresh flag to the index request and set it to true. Note though, that this will slow down indexing.
On Wednesday, March 16, 2011 at 12:36 AM, Boaz Leskes wrote:

Hi all,

I'm researching the possibility to use ES as our primary search engine
and I'm running into a couple of issues. In my scenario the documents
look something like this (for example):
{
id: 1
some_text_fields: "some text"
some_meta_numbers : [ 1,2,3,]
}

When a new document is added to the index it is OK it will only be
visible through search after a minute. However, sometimes the numbers
in the some_meta_numbers change and I need this change to be
immediately visible. In other words, if a search is done after these
numbers have been modified, it should immediately reflect the change.
For example for all documents have 1 as number should not return a
document whose number 1 has been remove a couple of milliseconds
before.

Is this possible?

Thanks for your help,
Boaz

3

Thanks.

I'm OK with slowing down background indexing of new documents, but
will this scale for say processing 50 changes to the meta numbers a
second on a single shard?

Boaz

On Mar 16, 12:18 am, Shay Banon shay.ba...@elasticsearch.com wrote:

It is possible, you can add a refresh flag to the index request and set it to true. Note though, that this will slow down indexing.

On Wednesday, March 16, 2011 at 12:36 AM, Boaz Leskes wrote:

Hi all,

I'm researching the possibility to use ES as our primary search engine
and I'm running into a couple of issues. In my scenario the documents
look something like this (for example):
{
id: 1
some_text_fields: "some text"
some_meta_numbers : [ 1,2,3,]
}

When a new document is added to the index it is OK it will only be
visible through search after a minute. However, sometimes the numbers
in the some_meta_numbers change and I need this change to be
immediately visible. In other words, if a search is done after these
numbers have been modified, it should immediately reflect the change.
For example for all documents have 1 as number should not return a
document whose number 1 has been remove a couple of milliseconds
before.

Is this possible?

Thanks for your help,
Boaz

4

Hard to tell, depends on a lot of factors such as machine type, number of docs. You will need to test it.
On Wednesday, March 16, 2011 at 3:45 PM, Boaz Leskes wrote:

Thanks.

I'm OK with slowing down background indexing of new documents, but
will this scale for say processing 50 changes to the meta numbers a
second on a single shard?

Boaz

On Mar 16, 12:18 am, Shay Banon shay.ba...@elasticsearch.com wrote:

It is possible, you can add a refresh flag to the index request and set it to true. Note though, that this will slow down indexing.

On Wednesday, March 16, 2011 at 12:36 AM, Boaz Leskes wrote:

Hi all,

I'm researching the possibility to use ES as our primary search engine
and I'm running into a couple of issues. In my scenario the documents
look something like this (for example):
{
id: 1
some_text_fields: "some text"
some_meta_numbers : [ 1,2,3,]
}

When a new document is added to the index it is OK it will only be
visible through search after a minute. However, sometimes the numbers
in the some_meta_numbers change and I need this change to be
immediately visible. In other words, if a search is done after these
numbers have been modified, it should immediately reflect the change.
For example for all documents have 1 as number should not return a
document whose number 1 has been remove a couple of milliseconds
before.

Is this possible?

Thanks for your help,
Boaz