ExternalFileField-like functionality

Elastic Stack Elasticsearch
1

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (see http://search-lucene.com/q=ExternalFileField )
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers -- http://sematext.com/about/jobs.html

2

Heya Otis,

Yes, there are plans to try and support something similar, but quite

different :slight_smile: in how its going to work than the external field feature (not a
fan of the it, personally). Note though, its a big feature, so it will take
some time to make it happen.

On Tue, Aug 16, 2011 at 2:54 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (see http://search-lucene.com/q=ExternalFileField )
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers -- Jobs - Sematext

3

I've looked into ExternalFileField in Solr and couldn't figure out a
way to make it work efficently.

The problem is that all those int/float values need to be stored in a
single file. So if I have 100 million documents, I'd need to have a
file with 100 million doc_id/value pairs in it. Then if I want to
update the value of just 1 document, how do I update that external
file? Do I go to the database and export the entire 100 million id/v
pairs to a new external file that overwrite the old file? That would
be extremely inefficient.

Otis -
Can you explain how did you update the external file when you used
ExternalFileField?

Shay -
Can you share a bit about what you have in mind? Hopefully it wouldn't
be too similar to ExternalFileField. Ideally it'd be just like
updating a RDBMS where I can atomically update/increment a specific
field without having to use external file. Don't know if that's
technically possible...

Andy

On Aug 16, 8:12 am, Shay Banon kim...@gmail.com wrote:

Heya Otis,

Yes, there are plans to try and support something similar, but quite

different :slight_smile: in how its going to work than the external field feature (not a
fan of the it, personally). Note though, its a big feature, so it will take
some time to make it happen.

On Tue, Aug 16, 2011 at 2:54 PM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (seehttp://search-lucene.com/q=ExternalFileField)
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers --Jobs - Sematext

4

Andy, you nailed the problem with external field, and thats not talking
about how to make it work with a dynamic distributed environment. I have a
different model in mind, but, its pretty rough and not complete, hard to
share it now... :slight_smile:

On Wed, Aug 17, 2011 at 4:20 AM, Andy selforganized@gmail.com wrote:

I've looked into ExternalFileField in Solr and couldn't figure out a
way to make it work efficently.

The problem is that all those int/float values need to be stored in a
single file. So if I have 100 million documents, I'd need to have a
file with 100 million doc_id/value pairs in it. Then if I want to
update the value of just 1 document, how do I update that external
file? Do I go to the database and export the entire 100 million id/v
pairs to a new external file that overwrite the old file? That would
be extremely inefficient.

Otis -
Can you explain how did you update the external file when you used
ExternalFileField?

Shay -
Can you share a bit about what you have in mind? Hopefully it wouldn't
be too similar to ExternalFileField. Ideally it'd be just like
updating a RDBMS where I can atomically update/increment a specific
field without having to use external file. Don't know if that's
technically possible...

Andy

On Aug 16, 8:12 am, Shay Banon kim...@gmail.com wrote:

Heya Otis,

Yes, there are plans to try and support something similar, but quite

different :slight_smile: in how its going to work than the external field feature
(not a
fan of the it, personally). Note though, its a big feature, so it will
take
some time to make it happen.

On Tue, Aug 16, 2011 at 2:54 PM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (seehttp://search-lucene.com/q=ExternalFileField)
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers --
Jobs - Sematext

5

It's great if ES has similar feature!

在 2011年8月16日星期二UTC+8下午7时54分02秒，Otis Gospodnetic写道：

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (see http://search-lucene.com/q=ExternalFileField )
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers -- Jobs - Sematext

在 2011年8月16日星期二UTC+8下午7时54分02秒，Otis Gospodnetic写道：

Hi,

In our engagements with our Solr customers we've successfully used
ExternalFileField (see http://search-lucene.com/q=ExternalFileField )
to handle situations where a numeric/float field needed to be updated
frequently while avoiding having to update the whole document. I'm
wondering what functionality one can use in ES to achieve the same
effect?

Thanks,
Otis

Sematext is hiring Search Engineers -- Jobs - Sematext

6

Elastic search has any alternate solution for this? In our project we use the external fields to store page views and some other frequently updating data and use that value to sort the overall result, its works perfect in solr, but now I am migrating our entire solr infrastructure to ES. Please guide me, how can i replace external file fields in ES...

7

For page views, use doc values, like

"page_views": {
  "type": "integer",
  "index": "no",
  "fielddata": {
    "format": "doc_values"
  }
}

See

8

But I don't want to update the index every 5 10 mts, that may slow down queries to the same index right? That's the reason I was keeping it in external file field.

Do you think it will be a issue when I update the index while other clients querying the same index?