Intercept transport client/http client actions


(Mustafa Sener) #1

Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com


(Shay Banon) #2

This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).

The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?

-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com


(Mustafa Sener) #3

No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).

The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?

-shay.banon

On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Shay Banon) #4

Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:

This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).

The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?

-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Mustafa Sener) #5

I created the issue


can you explain about internal mechanism? How can we use it?

On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.

On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).

The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?

-shay.banon

On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Shay Banon) #6

Its not exposed as an API (yet?), its used internally in elasticsearch (to block operations till recovery has happened for example). But, by design, it can easily be exposed as a public API for custom "blocks" to be defined per index. So, for example, you would have an API to add a block to index and block writes.
On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:

I created the issue
https://github.com/elasticsearch/elasticsearch/issues/785
can you explain about internal mechanism? How can we use it?

On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:

This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).

The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?

-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Mustafa Sener) #7

It will be a very good feature

On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Its not exposed as an API (yet?), its used internally in elasticsearch
(to block operations till recovery has happened for example). But, by
design, it can easily be exposed as a public API for custom "blocks" to be
defined per index. So, for example, you would have an API to add a block to
index and block writes.

On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:

I created the issue
https://github.com/elasticsearch/elasticsearch/issues/785
can you explain about internal mechanism? How can we use it?

On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.

On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.comwrote:

This is certainly possible, though at a cost. Th proper place to do that
is actually on the primary shard using some sort of callback (so you know it
only happens once).

The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?

-shay.banon

On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Shay Banon) #8

Open an issue for this as well, so we won't forget.
On Wednesday, March 16, 2011 at 11:21 PM, Mustafa Sener wrote:

It will be a very good feature

On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Its not exposed as an API (yet?), its used internally in elasticsearch (to block operations till recovery has happened for example). But, by design, it can easily be exposed as a public API for custom "blocks" to be defined per index. So, for example, you would have an API to add a block to index and block writes.
On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:

I created the issue
https://github.com/elasticsearch/elasticsearch/issues/785
can you explain about internal mechanism? How can we use it?

On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon shay.banon@elasticsearch.com wrote:

Got you. Actually, for blocking, there is already an internally used mechanism to block operations on indices (READ, WRITE, and META), it just needs to be exposed as a public API.
On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to block write operations for a certain time. This interception mechanism will be useful for this purpose too. I understood the cost of deserializing. Maybe we need to think again for enrichment. But it will be a very good improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon shay.banon@elasticsearch.com wrote:

This is certainly possible, though at a cost. Th proper place to do that is actually on the primary shard using some sort of callback (so you know it only happens once).

The cost associated is the fact that the document source will need to be parsed and represented in memory, then allow to callback to munge it, and then serialized again. This is because the way elasticsearch parses the document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do? Will you need to RPC call other systems to do it?

-shay.banon
On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to intercept action such as index to make some modifications on data before inserted. There will be many client around ES and we don't want to take care of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(Mustafa Sener) #9

I created the issue

On Wed, Mar 16, 2011 at 11:31 PM, Shay Banon
shay.banon@elasticsearch.comwrote:

Open an issue for this as well, so we won't forget.

On Wednesday, March 16, 2011 at 11:21 PM, Mustafa Sener wrote:

It will be a very good feature

On Wed, Mar 16, 2011 at 11:20 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

Its not exposed as an API (yet?), its used internally in elasticsearch
(to block operations till recovery has happened for example). But, by
design, it can easily be exposed as a public API for custom "blocks" to be
defined per index. So, for example, you would have an API to add a block to
index and block writes.

On Wednesday, March 16, 2011 at 11:18 PM, Mustafa Sener wrote:

I created the issue
https://github.com/elasticsearch/elasticsearch/issues/785
can you explain about internal mechanism? How can we use it?

On Wed, Mar 16, 2011 at 11:13 PM, Shay Banon <
shay.banon@elasticsearch.com> wrote:

Got you. Actually, for blocking, there is already an internally used
mechanism to block operations on indices (READ, WRITE, and META), it just
needs to be exposed as a public API.

On Wednesday, March 16, 2011 at 11:11 PM, Mustafa Sener wrote:

No we will not use RPC. We need this to enrich the data. Also we want to
block write operations for a certain time. This interception mechanism will
be useful for this purpose too. I understood the cost of deserializing.
Maybe we need to think again for enrichment. But it will be a very good
improvement for other needs.

On Wed, Mar 16, 2011 at 9:39 PM, Shay Banon <shay.banon@elasticsearch.com

wrote:

This is certainly possible, though at a cost. Th proper place to do
that is actually on the primary shard using some sort of callback (so you
know it only happens once).

The cost associated is the fact that the document source will need to be
parsed and represented in memory, then allow to callback to munge it, and
then serialized again. This is because the way elasticsearch parses the
document is using pull parsing.

Open an issue for that. Also, what kind of changes are you going to do?
Will you need to RPC call other systems to do it?

-shay.banon

On Wednesday, March 16, 2011 at 9:57 AM, Mustafa Sener wrote:

Hi,
Is it possible to intercept transport client actions? Basically we want to
intercept action such as index to make some modifications on data before
inserted. There will be many client around ES and we don't want to take care
of this in every client. This will be beneficial for some other reasons too.

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com

--
Mustafa Sener
www.ifountain.com


(system) #10