in the last couple of years we managed to customize Apache Lucene (through
its public API) to support branching, tagging and compare in a concurrent
fashion for our server application. We managed to achieve this by using a
couple of custom Directory, exactly one IndexDeletionPolicy and one MergePolicy
implementations. Currently we are considering to replace Lucene with
Elasticsearch on the server-side. Before we jumped into the details of
collecting the differences between the two technologies in respect of the
search and indexing functionality and for instance how to port our custom
collectors and how to replace NDVs, we would like to make sure if it is
possible at all.
I've just checked out the source and realized that the registration of the
services are done via various module implementations and the actual
configured service implementations are injected into the constructors. For
the sake of simplicity is there a way for example to create an
Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy
instead of the default KeepOnlyLastDeletionPolicy? I assume if this is
straightforward we could use or custom implementations for the directory
and the IndexWriter what we are currently using with pure Lucene. After
doing some research I found this 1 thread. Am I close to the
answer/solution?
I have to notice we would like to achieve this without forking the public
repository.
I know the merge policy is configurable but I don't know if it is
pluggable. I imagine it'd be pretty simple to make what you need pluggable
if it isn't already. You'd have to send a pull request but you wouldn't
have to maintain the fork for now than a release I imagine.
On Oct 27, 2014 8:22 AM, "Ákos Kitta" kittaakos@gmail.com wrote:
Hi there,
in the last couple of years we managed to customize Apache Lucene (through
its public API) to support branching, tagging and compare in a concurrent
fashion for our server application. We managed to achieve this by using a
couple of custom Directory, exactly one IndexDeletionPolicy and one MergePolicy
implementations. Currently we are considering to replace Lucene with
Elasticsearch on the server-side. Before we jumped into the details of
collecting the differences between the two technologies in respect of the
search and indexing functionality and for instance how to port our custom
collectors and how to replace NDVs, we would like to make sure if it is
possible at all.
I've just checked out the source and realized that the registration of the
services are done via various module implementations and the actual
configured service implementations are injected into the constructors. For
the sake of simplicity is there a way for example to create an
Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy
instead of the default KeepOnlyLastDeletionPolicy? I assume if this is
straightforward we could use or custom implementations for the directory
and the IndexWriter what we are currently using with pure Lucene. After
doing some research I found this 1 thread. Am I close to the
answer/solution?
I have to notice we would like to achieve this without forking the public
repository.
Regarding the deletion policy, you can set the class name of your deletion
policy implementation in the setting "index.deletionpolicy.type"
For custom Directory, you have to
patch org.elasticsearch.index.store.IndexStoreModule with your custom index
store. The index store is something like an IndexWriter / Lucene Directory
on steroids. At the moment, it is not possible to add custom index stores
from a plugin (see the fixed enumeration of implementations
in IndexStoreModule)
in the last couple of years we managed to customize Apache Lucene (through
its public API) to support branching, tagging and compare in a concurrent
fashion for our server application. We managed to achieve this by using a
couple of custom Directory, exactly one IndexDeletionPolicy and one MergePolicy
implementations. Currently we are considering to replace Lucene with
Elasticsearch on the server-side. Before we jumped into the details of
collecting the differences between the two technologies in respect of the
search and indexing functionality and for instance how to port our custom
collectors and how to replace NDVs, we would like to make sure if it is
possible at all.
I've just checked out the source and realized that the registration of the
services are done via various module implementations and the actual
configured service implementations are injected into the constructors. For
the sake of simplicity is there a way for example to create an
Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy
instead of the default KeepOnlyLastDeletionPolicy? I assume if this is
straightforward we could use or custom implementations for the directory
and the IndexWriter what we are currently using with pure Lucene. After
doing some research I found this 1 thread. Am I close to the
answer/solution?
I have to notice we would like to achieve this without forking the public
repository.
I stand corrected - there is also the possibility of the setting
"index.store.type", by setting this to a Java class name, you can use this
as the current index store implementation from a plugin.
Regarding the deletion policy, you can set the class name of your deletion
policy implementation in the setting "index.deletionpolicy.type"
For custom Directory, you have to
patch org.elasticsearch.index.store.IndexStoreModule with your custom index
store. The index store is something like an IndexWriter / Lucene Directory
on steroids. At the moment, it is not possible to add custom index stores
from a plugin (see the fixed enumeration of implementations
in IndexStoreModule)
in the last couple of years we managed to customize Apache Lucene
(through its public API) to support branching, tagging and compare in a
concurrent fashion for our server application. We managed to achieve this
by using a couple of custom Directory, exactly one IndexDeletionPolicy and
one MergePolicy implementations. Currently we are considering to replace
Lucene with Elasticsearch on the server-side. Before we jumped into the
details of collecting the differences between the two technologies in
respect of the search and indexing functionality and for instance how to
port our custom collectors and how to replace NDVs, we would like to make
sure if it is possible at all.
I've just checked out the source and realized that the registration of
the services are done via various module implementations and the actual
configured service implementations are injected into the constructors. For
the sake of simplicity is there a way for example to create an
Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy
instead of the default KeepOnlyLastDeletionPolicy? I assume if this is
straightforward we could use or custom implementations for the directory
and the IndexWriter what we are currently using with pure Lucene. After
doing some research I found this 1 thread. Am I close to the
answer/solution?
I have to notice we would like to achieve this without forking the public
repository.
Awesome. Thanks a lot for the help. I'll give a try.
On Monday, October 27, 2014 2:30:51 PM UTC+1, Jörg Prante wrote:
I stand corrected - there is also the possibility of the setting
"index.store.type", by setting this to a Java class name, you can use this
as the current index store implementation from a plugin.
Regarding the deletion policy, you can set the class name of your
deletion policy implementation in the setting "index.deletionpolicy.type"
For custom Directory, you have to
patch org.elasticsearch.index.store.IndexStoreModule with your custom index
store. The index store is something like an IndexWriter / Lucene Directory
on steroids. At the moment, it is not possible to add custom index stores
from a plugin (see the fixed enumeration of implementations
in IndexStoreModule)
Jörg
On Mon, Oct 27, 2014 at 1:22 PM, Ákos Kitta <kitt...@gmail.com
<javascript:>> wrote:
Hi there,
in the last couple of years we managed to customize Apache Lucene
(through its public API) to support branching, tagging and compare in a
concurrent fashion for our server application. We managed to achieve this
by using a couple of custom Directory, exactly one IndexDeletionPolicy and
one MergePolicy implementations. Currently we are considering to
replace Lucene with Elasticsearch on the server-side. Before we jumped into
the details of collecting the differences between the two technologies in
respect of the search and indexing functionality and for instance how to
port our custom collectors and how to replace NDVs, we would like to make
sure if it is possible at all.
I've just checked out the source and realized that the registration of
the services are done via various module implementations and the actual
configured service implementations are injected into the constructors. For
the sake of simplicity is there a way for example to create an
Elasticsearch module which forces the underlying IndexWriter to use the FooCustomDeletionPolicy
instead of the default KeepOnlyLastDeletionPolicy? I assume if this is
straightforward we could use or custom implementations for the directory
and the IndexWriter what we are currently using with pure Lucene. After
doing some research I found this 1 thread. Am I close to the
answer/solution?
I have to notice we would like to achieve this without forking the
public repository.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.