Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.
--
Mustafa Sener
www.ifountain.com
This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).
The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.
Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?
-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:
Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com
No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.
On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:
This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:
Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com
--
Mustafa Sener
www.ifountain.com
Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:
No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.
On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:
This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).
The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.
Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?
-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
I created the issue
can you explain about internal mechanism? How can we use it?
On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon
shay.banon@elasticsearch.comwrote:
Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:
No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:
This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:
Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
--
Mustafa Sener
www.ifountain.com
Its not exposed as an API (yet?), its used internally in elasticsearch (to block operations till recovery has happened for example). But, by design, it can easily be exposed as a public API for custom "blocks" to be defined per index. So, for example, you would have an API to add a block to index and block writes.
On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:
I created the issue
A mechanism to intercept transport client/http client actions · Issue #785 · elastic/elasticsearch · GitHub
can you explain about internal mechanism? How can we use it?On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon shay.banon@elasticsearch.com wrote:
Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.
On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:
This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).
The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.
Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?
-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
It will be a very good feature
On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon
shay.banon@elasticsearch.comwrote:
Its not exposed as an API (yet?), its used internally in elasticsearch
(to block operations till recovery has happened for example). But, by
design, it can easily be exposed as a public API for custom "blocks" to be
defined per index. So, for example, you would have an API to add a block to
index and block writes.On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:
I created the issue
A mechanism to intercept transport client/http client actions · Issue #785 · elastic/elasticsearch · GitHub
can you explain about internal mechanism? How can we use it?On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon <shay.banon@elasticsearch.com
wrote:
Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:
No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:
This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:
Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
--
Mustafa Sener
www.ifountain.com
Open an issue for this as well, so we won't forget.
On Wednesday, March 16, 2011 at 11:21 PM, Mustafa Sener wrote:
It will be a very good feature
On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon shay.banon@elasticsearch.com wrote:
Its not exposed as an API (yet?), its used internally in elasticsearch (to block operations till recovery has happened for example). But, by design, it can easily be exposed as a public API for custom "blocks" to be defined per index. So, for example, you would have an API to add a block to index and block writes.
On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:I created the issue
A mechanism to intercept transport client/http client actions · Issue #785 · elastic/elasticsearch · GitHub
can you explain about internal mechanism? How can we use it?On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon shay.banon@elasticsearch.com wrote:
Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.
On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:
This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).
The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.
Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?
-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
I created the issue
On Wed, Mar 16, 2011 at 11:31 PM, Shay Banon
shay.banon@elasticsearch.comwrote:
Open an issue for this as well, so we won't forget.
On Wednesday, March 16, 2011 at 11:21 PM, Mustafa Sener wrote:
It will be a very good feature
On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon <shay.banon@elasticsearch.com
wrote:
Its not exposed as an API (yet?), its used internally in elasticsearch
(to block operations till recovery has happened for example). But, by
design, it can easily be exposed as a public API for custom "blocks" to be
defined per index. So, for example, you would have an API to add a block to
index and block writes.On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:
I created the issue
A mechanism to intercept transport client/http client actions · Issue #785 · elastic/elasticsearch · GitHub
can you explain about internal mechanism? How can we use it?On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:
No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon <shay.banon@elasticsearch.com
wrote:
This is certainly possible, though at a cost. Th proper place to do
that is actually on the primary shard using some sort of callback (so you
know it only happens once).The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:
Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com--
Mustafa Sener
www.ifountain.com
--
Mustafa Sener
www.ifountain.com
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.