I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type of
problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
If you are only modifying the REST API calls and not the Java API, such a
plugin should be easy. You are not creating a new type of action, merely
using the current search one, but changing the output format.
Hopefully the content is not too old. Base the plugin around the
existing RestSearchAction, but in the handleRequest method, instead of
returning the results directly, you can modify them before.
Cheers
Ivan
On Thu, Jun 5, 2014 at 10:45 AM, Mario Mueller mario@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
So, if I understood your approach in the right way ... I should build a new
Rest Action like "_search_and_return_source" that proxies the original
_search one?
I've already read those two articles and I've set up my development
environment with the help of those
Am Donnerstag, 5. Juni 2014 19:55:06 UTC+2 schrieb Ivan Brusic:
If you are only modifying the REST API calls and not the Java API, such a
plugin should be easy. You are not creating a new type of action, merely
using the current search one, but changing the output format.
Hopefully the content is not too old. Base the plugin around the
existing RestSearchAction, but in the handleRequest method, instead of
returning the results directly, you can modify them before.
Cheers
Ivan
On Thu, Jun 5, 2014 at 10:45 AM, Mario Mueller <ma...@xenji.com
<javascript:>> wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Just a quick question, do you just want to extract a field from the json
source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller mario@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the json
source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller <ma...@xenji.com
<javascript:>> wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the json
source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the json
source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this
type of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if
so, what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
I just looked it up and it should be as easy as creating your
own RestResponseListener that takes a SearchResponse and creates a
simplified version with no metadata.
Should be an interesting quick plugin, but it looks like Jorg is going to
beat me to it (I'm still at work for several more hours).
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the
json source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this
type of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if
so, what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
On Thu, Jun 5, 2014 at 10:28 PM, Ivan Brusic ivan@brusic.com wrote:
I just looked it up and it should be as easy as creating your
own RestResponseListener that takes a SearchResponse and creates a
simplified version with no metadata.
Should be an interesting quick plugin, but it looks like Jorg is going to
beat me to it (I'm still at work for several more hours).
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the
json source?
There are field filters and parameters for shaping such a JSON result,
maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this
type of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if
so, what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
I see that we agree that a new RestResponseListener is the way to go.
I have not cloned your project yet, only looked at the code on github, but
I noticed that you provided your own parseSearchRequest, but still
call RestSearchAction.parseSearchRequest from inside handleRequest. Did I
misinterpret the code or is that a mistake?
On Thu, Jun 5, 2014 at 10:28 PM, Ivan Brusic ivan@brusic.com wrote:
I just looked it up and it should be as easy as creating your
own RestResponseListener that takes a SearchResponse and creates a
simplified version with no metadata.
Should be an interesting quick plugin, but it looks like Jorg is going to
beat me to it (I'm still at work for several more hours).
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the
json source?
There are field filters and parameters for shaping such a JSON
result, maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com
wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this
type of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if
so, what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
Ups, yes, a mistake... I bluntly copy/pasted the RestSearchAction. Thanks!
Jörg
On Fri, Jun 6, 2014 at 12:03 AM, Ivan Brusic ivan@brusic.com wrote:
I see that we agree that a new RestResponseListener is the way to go.
I have not cloned your project yet, only looked at the code on github, but
I noticed that you provided your own parseSearchRequest, but still
call RestSearchAction.parseSearchRequest from inside handleRequest. Did I
misinterpret the code or is that a mistake?
On Thu, Jun 5, 2014 at 10:28 PM, Ivan Brusic ivan@brusic.com wrote:
I just looked it up and it should be as easy as creating your
own RestResponseListener that takes a SearchResponse and creates a
simplified version with no metadata.
Should be an interesting quick plugin, but it looks like Jorg is going
to beat me to it (I'm still at work for several more hours).
Those fields are not stored, but indexed. As I mentioned before, this
source matches exactly to an entity structure in our application and we
need the structure of the source document 1:1. If this is achievable by any
other methods than a plugin, then I am totally fine with it.
Am Donnerstag, 5. Juni 2014 21:36:31 UTC+2 schrieb Jörg Prante:
Just a quick question, do you just want to extract a field from the
json source?
There are field filters and parameters for shaping such a JSON
result, maybe they can already help?
Or can you give an example of the problem?
Jörg
On Thu, Jun 5, 2014 at 7:45 PM, Mario Mueller ma...@xenji.com
wrote:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this
type of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And
if so, what type of plugin? I've looked at the RestFilter, but I don't know
if this is the right way to go...
Any hints are welcome!
Regards,
Mario
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
This may or may not help, but the following worked well for me. Just as any
database-backed application, the business logic (such as what you
described) is best implemented outside of the database. Since ES is a
first-class Java citizen and its Java API is clean and superb
(documentation shortcomings are more than made up for by the excellent
advice from this newsgroup), this is what I did:
Create my business logic, and use the TransportClient to contact
Elasticsearch. It's easy to test it in a stand-alone driver.
Create a server using Netty and the LMAX Disruptor. Netty is awesome,
and I could craft the interface to accept the questions that made sense to
the business, not to Elasticsearch. Then I instantiated an LMAX Disruptor
in multi-producer mode, and each Netty thread simply created an event that
contained 2 objects: The decoded HTTP request parameter, and the reference
to the response output channel.
The Disruptor contained a fixed number of Worker handler threads (to
match those in Elasticsearch) that actually converted the request into one
or more Elasticsearch requests, performing whatever functions the business
logic required, and then writing the responses back to the response channel.
Even if my server got a huge burst of requests, they were quickly queued
which means that the Netty handler thread count stays low. And then the
back-end Disruptor thread max will ensure that Elasticsearch is never
overloaded.
This is kind of the hard way using Java, but it suited me fine. Other
solutions could be similarly implemented with comperable performance but
perhaps a lot less development time in Python with Django, or Node.js, or
Ruby on Rails. But Java worked well for me; Netty is fast and easy to work
with, and the LMAX Disruptor offers a near-zero latency nearly lockless
queuing, and together they keep the thread count low while still handling
many many more requests at one time.
This sounds like a lot of work, but the hard work is a one-off kind of
thing, and then the business logic can be easily tuned, fixed, and extended
without much effort. Even things that Elasticsearch does not do and will
never do (such as foreign keys and enforced consistency across types or
even indices) are relatively easily done within the custom business logic
of a front-end server.
Again, I don't offer this as a strong suggestion but rather as a case study
of what worked well for me.
You guys are totally awesome! Thanks a lot! If you ever visit Duesseldorf
drop me a line, I owe you a beer.
@Brian:
Interesting approach, but wouldn't this go against the initial "no
additional proxy" statement, if I got you right ..
Am Donnerstag, 5. Juni 2014 19:45:33 UTC+2 schrieb Mario Mueller:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
I drink Kölsch only ävver et hätt noh immer joot jejange
Greetings from Cologne!
Jörg
On Fri, Jun 6, 2014 at 7:14 AM, Mario Mueller mario@xenji.com wrote:
You guys are totally awesome! Thanks a lot! If you ever visit Duesseldorf
drop me a line, I owe you a beer.
@Brian:
Interesting approach, but wouldn't this go against the initial "no
additional proxy" statement, if I got you right ..
Am Donnerstag, 5. Juni 2014 19:45:33 UTC+2 schrieb Mario Mueller:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the client.
The php app that sits on the other side uses JMS\Serializer to deserialize
the response into entities. At the moment the app needs to take an overhead
to derserialize it, extract the source and serialize it again. Then the
serialized stuff is passed to the entity deserializer. That's really
painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Et kütt wie et kütt Das mit dem Koelsch geht klar, gibts auch hier in
DUS.
Again thanks to all!
Am Freitag, 6. Juni 2014 09:09:39 UTC+2 schrieb Jörg Prante:
I drink Kölsch only ävver et hätt noh immer joot jejange
Greetings from Cologne!
Jörg
On Fri, Jun 6, 2014 at 7:14 AM, Mario Mueller <ma...@xenji.com
<javascript:>> wrote:
You guys are totally awesome! Thanks a lot! If you ever visit Duesseldorf
drop me a line, I owe you a beer.
@Brian:
Interesting approach, but wouldn't this go against the initial "no
additional proxy" statement, if I got you right ..
Am Donnerstag, 5. Juni 2014 19:45:33 UTC+2 schrieb Mario Mueller:
Hey folks,
I kindly ask for a hint to achieve the following thing:
The goal is to deliver only a json array of source objects to the
client. The php app that sits on the other side uses JMS\Serializer to
deserialize the response into entities. At the moment the app needs to take
an overhead to derserialize it, extract the source and serialize it again.
Then the serialized stuff is passed to the entity deserializer. That's
really painful.
I've found a thread that suggests a proxy in between to handle this type
of problem, but this is not possible in our env.
The real question is: Is this achievable by writing a plugin? And if so,
what type of plugin? I've looked at the RestFilter, but I don't know if
this is the right way to go...
Hi, Mario. Yes, I suppose this kind of goes against the "no additional
proxy" requirement you have.
Hehehe. I'm a seeker of loopholes. In my scenario, it's still a plug-in
design, but ES is my plug-in and not the other way around. Still only one
HTTP interface in the mix, but it's mine and not ES's.
I also have avoided the plug-in approach because I've read that it's marked
for deprecation and eventual removal. Yet logstash and ES Head are still
offered as plug-ins as are a boat-load of other facilities, so I am not
really sure if that's still the case.
And of course, your own plug-in has a much better chance to be updated to
match exactly each new ES version to which you migrate. That's one of the
downsides of third-party plug-ins: They lock you into older ES versions
until the author gets a chance to update the plug-in.
Brian
On Friday, June 6, 2014 1:14:00 AM UTC-4, Mario Mueller wrote:
@Brian:
Interesting approach, but wouldn't this go against the initial "no
additional proxy" statement, if I got you right ..
Plugins are essential to ES's success and are not going away any time soon.
The river plugins, aka cluster singletons, are the ones which are
discouraged from use. Good ahead and create more plugins!
I also have avoided the plug-in approach because I've read that it's
marked for deprecation and eventual removal. Yet logstash and ES Head are
still offered as plug-ins as are a boat-load of other facilities, so I am
not really sure if that's still the case.
Thanks so much, Ivan. That's a very important distinction.
Brian
On Friday, June 6, 2014 12:28:56 PM UTC-4, Ivan Brusic wrote:
Plugins are essential to ES's success and are not going away any time
soon. The river plugins, aka cluster singletons, are the ones which are
discouraged from use. Good ahead and create more plugins!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
ävver et hätt noh immer joot jejange