ElasticSearch while browsing an index that is updated every couple of minutes


(Tom K) #1

Hello,

To be clear and transparent - I know very little about ES. I would normally
lurk around, but figure I might as well ask.

I have a project that I think would be a good fit for ES, but wondering to
what extent. I'd like to use ES for what it is really good at vs trying to
make it work in other situations.

The project is www.lionseek.com

One part of me thinks that ES would be best if I used it only when a
visitor conducts a text search and thereby replacing a fulltext search via
mysql. The search query would tap into the ES index which updates every
minute or so and the results would be displayed.

The other part of me thinks that ES can be used for much more. Basically
every time a page is rendered, ES is involved in one way or another
regardless if a search term/parameter is entered.

A concern I do have (with my very very limited knowledge of ES) is how
relative timestamps would be used.

For a specific example:

  1. let's say that Lionseek indexes content from a forum
  2. it knows that the time of the posting is 9am today
  3. the timezone of the forum is in San Francisco (UTC-08:00)
  4. a time offset is entered
  5. a relative time is shown to the visitor - i.e. 'updated 2 hours ago'

In MySQL we would do a sum() and join. We'd sum up the offset and the
posted_time column as normalized_time.

In ES, it seems that can't be done? ES uses UTC as the timezone and that
seems unchangeable. Maybe a new field needs to be added?

Thanks for taking the time to read through this and my ultimate question in
the end is.... to what extent do you see ES being integrated into a project
like Lionseek? I know it's asking a lot, but I'm hoping the gurus here can
point to things they see and say... 'Well, I think ES would be an excellent
component to help you in [this] regard or [that] regard.'

-Tom

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/39943803-1918-4843-97ca-6c532925f47d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Binh Ly) #2

Tom,

You might be interested in the script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html

It allows you to introduce a dynamically computed field at query-time where
you can script the logic on how the field value is computed - for example
like doing time offset adjustments.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e403a680-078f-4363-b110-419243678ed2%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tom K) #3

Awesome - I could definitely see using the script for calculating the
adjustments. Thanks!

Any other ideas where ES should/should not be used on the site?

On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:

Tom,

You might be interested in the script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html

It allows you to introduce a dynamically computed field at query-time
where you can script the logic on how the field value is computed - for
example like doing time offset adjustments.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Nik Everett) #4

In past lives I was involved in integrating Solr to power all kinds of
funky little "shelves" on a page. Elasticsearch can handle it. The trick
is to make sure that you don't do too many slow thing against too many
documents. If the shelf is built by matches and sorting (by a field or
relevance) you are fine. Just don't run script scoring on tens of
thousands of results on every page hit.

Nik

On Fri, Feb 7, 2014 at 4:33 PM, Tom K tommyk@gmail.com wrote:

Awesome - I could definitely see using the script for calculating the
adjustments. Thanks!

Any other ideas where ES should/should not be used on the site?

On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:

Tom,

You might be interested in the script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/search-request-script-fields.html

It allows you to introduce a dynamically computed field at query-time
where you can script the logic on how the field value is computed - for
example like doing time offset adjustments.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1sh9_QThr8Vaa33EowYT7Wa5FEp4ob%2B39pA6U6z-kA5g%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Binh Ly) #5

You have sorting and paging which are very straightforward to implement in
ES. You can also do full text search using ES easily - for example, if I
type like "club snap" into the search box, you can run a single query that
looks for that text across multiple fields in ES easily. You can also
further enhance the search experience using ES facets so for example, give
hints to the user while he is searching, here are the top items by
popularity, or by price ranges, or by brand or something along those lines.
:slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23d39888-8b17-472e-9441-6627f03918d0%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tom K) #6

A number of categories (i.e. watches) has 100's of thousands of possible
results that ultimately need to be sorted - price/brand/time. The default
sorting will be by time. Are you saying that this type of sorting may be
difficult/slow? Nothing bugs me more than a slow site - that's one of the
main reasons I'm looking at ES.

On Friday, February 7, 2014 1:51:49 PM UTC-8, Nikolas Everett wrote:

In past lives I was involved in integrating Solr to power all kinds of
funky little "shelves" on a page. Elasticsearch can handle it. The trick
is to make sure that you don't do too many slow thing against too many
documents. If the shelf is built by matches and sorting (by a field or
relevance) you are fine. Just don't run script scoring on tens of
thousands of results on every page hit.

Nik

On Fri, Feb 7, 2014 at 4:33 PM, Tom K <tom...@gmail.com <javascript:>>wrote:

Awesome - I could definitely see using the script for calculating the
adjustments. Thanks!

Any other ideas where ES should/should not be used on the site?

On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:

Tom,

You might be interested in the script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/
reference/current/search-request-script-fields.html

It allows you to introduce a dynamically computed field at query-time
where you can script the logic on how the field value is computed - for
example like doing time offset adjustments.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/34d82190-618d-4849-bb74-d804cea7c066%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Tom K) #7

Good idea w/ the facets - I'll definitely be looking into that more! Thanks!

On Friday, February 7, 2014 1:54:33 PM UTC-8, Binh Ly wrote:

You have sorting and paging which are very straightforward to implement in
ES. You can also do full text search using ES easily - for example, if I
type like "club snap" into the search box, you can run a single query that
looks for that text across multiple fields in ES easily. You can also
further enhance the search experience using ES facets so for example, give
hints to the user while he is searching, here are the top items by
popularity, or by price ranges, or by brand or something along those lines.
:slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/571e1787-6158-4478-a64f-718f170826ce%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Nik Everett) #8

Sorting is quick, but you can obviously ask Elasticsearch to do slow
things. Just test things before you implement them on the data to be sure
that what you are doing isn't slow.

On Fri, Feb 7, 2014 at 5:16 PM, Tom K tommyk@gmail.com wrote:

A number of categories (i.e. watches) has 100's of thousands of possible
results that ultimately need to be sorted - price/brand/time. The default
sorting will be by time. Are you saying that this type of sorting may be
difficult/slow? Nothing bugs me more than a slow site - that's one of the
main reasons I'm looking at ES.

On Friday, February 7, 2014 1:51:49 PM UTC-8, Nikolas Everett wrote:

In past lives I was involved in integrating Solr to power all kinds of
funky little "shelves" on a page. Elasticsearch can handle it. The trick
is to make sure that you don't do too many slow thing against too many
documents. If the shelf is built by matches and sorting (by a field or
relevance) you are fine. Just don't run script scoring on tens of
thousands of results on every page hit.

Nik

On Fri, Feb 7, 2014 at 4:33 PM, Tom K tom...@gmail.com wrote:

Awesome - I could definitely see using the script for calculating the
adjustments. Thanks!

Any other ideas where ES should/should not be used on the site?

On Friday, February 7, 2014 5:10:36 AM UTC-8, Binh Ly wrote:

Tom,

You might be interested in the script_fields functionality:

http://www.elasticsearch.org/guide/en/elasticsearch/referenc
e/current/search-request-script-fields.html

It allows you to introduce a dynamically computed field at query-time
where you can script the logic on how the field value is computed - for
example like doing time offset adjustments.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/6d1d74e0-6341-4892-9bf4-e05bece740c9%
40googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/34d82190-618d-4849-bb74-d804cea7c066%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd23oUjbdOff7XZxUtE4xPQHbGJRMHD74PcWLTS2S0B5aw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9