We have recently migrated our application from 'bare Lucene + Zoie for
realtime search' to Elastic Search. Elastic search is awesome and next to
scalability, it gives us lots of additional features. The one thing we
really miss though is realtime search.
Search is the core of our application. All our data is stored in the index
(primary data store). When a user adds a file or makes a change, their
subsequent search must reflect that change. With Zoie, the data was indexed
very quickly into a temporary Lucene memory index. Not having to write+read
it on disk makes the documents available for search much faster than NRT
Lucene. The memory index is flushed to disk asynchrounously from time to
time, not impacting indexing or search performance. Zoie also allows you to
wait for a specific 'version of the index' to be available for searching.
That way we could make the user's thread wait until their data was indexed
in memory, only pausing the thread of that user without having any
performance impact for all the other users.
Result: realtime search and insanely fast indexing.
With Elastic Search we have to do a refresh to make data available for
search. Lots of refreshes or the 1 second refresh interval will cause
significant slower indexing speed. We don't know beforehand when our users
will import documents or make lots of changes, so we cannot really increase
the refresh interval when needed to make indexing faster. We know that
'get' is realtime and we make use of that as much as possible, but in lots
of cases we really require a search to find the data.
Our plan is to implement some mechanism in Elastic Search to get the same
realtime search + fast indexing behavior that we had with Zoie. We need
some pointers though on what would be the best place in Elastic Search to
do something like this. After all it hooks into low level Elastic Search
and Lucene stuff.
I can imagine that 'realtime-search while indexing' is important for many
other Elastic Search users too. What are the chances of something like this
getting merged back into the main branch?
I'm planning to be at the Friday drinks tomorrow in Amsterdam. Is there
anyone attending with whom I could do some sparring with on this matter?
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firstname.lastname@example.org.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ed50d5f-4ade-4d56-af06-6e2c26feff9b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.