ElasticSearch and Wicket -> serialization nightmare


(K.B.) #1

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(Shay Banon) #2

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...
On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(K.B.) #3

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte[] buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte[]) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(Shay Banon) #4

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).
On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte[] buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte[]) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(K.B.) #5

I currently pass a SearchSourceBilder around and manipulate it at
certain places. However I need to be able to persist the result of the
manipulations inside a class that wants it content to be serializable
(no further manipulation afterwards beside paging);
Now, the bye[] from sourceBuilder is the whole query, right? Is there
any way to send this to ES and only manipulate
the .setFrom(int).SetSize(int) at it?

Or what else could be passed around to later recreate a new
SearchRequestBuilder based upon it (I saw that SearchRequestBuilder
accepts Query, Facet and Filter as byte[])

Thanks for keeping up with me :slight_smile:

On 30 Mrz., 20:03, Shay Banon shay.ba...@elasticsearch.com wrote:

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).

On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte[] buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte[]) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(K.B.) #6

In case other have similar problems:

Solution is to build the SearchSourceBuilder as bytes[];

one so can do:

this.sourceBuilder = sourceBuilder.buildAsBytes(XContentType.SMILE);

and have

private byte[] sourceBuilder;

on class level;

Search then can be used like:

client().search(new SearchRequest(INDEX, byte[]
sourceBuilder).extraSource(any extra byte[] extraSource)

so a precompiled search can be narrowed or limited to certain results;

e.g:

getClient().search(new
SearchRequest(searchindex).source(source).extraSource(extraSource)).actionGet()

this now allows fast and very nice queries against any constructed
query as well as storing for later use; One so can build DataProviders
for wicket, historize search-queries for easy use in search
breadcrumbing, complete transparent backbutton support, etc.;
Performance didn't seem to get any big impact;

Any comments appreciated;

On 30 Mrz., 20:18, "K.B." korbinian.ba...@googlemail.com wrote:

I currently pass a SearchSourceBilder around and manipulate it at
certain places. However I need to be able to persist the result of the
manipulations inside a class that wants it content to be serializable
(no further manipulation afterwards beside paging);
Now, the bye[] from sourceBuilder is the whole query, right? Is there
any way to send this to ES and only manipulate
the .setFrom(int).SetSize(int) at it?

Or what else could be passed around to later recreate a new
SearchRequestBuilder based upon it (I saw that SearchRequestBuilder
accepts Query, Facet and Filter as byte[])

Thanks for keeping up with me :slight_smile:

On 30 Mrz., 20:03, Shay Banon shay.ba...@elasticsearch.com wrote:

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).

On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte[] buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte[]) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best


(Karussell) #7

Hi,

there. I'm using a separate persistable object in my wicket app:

or a bit easier:

Regards,
Peter.


(K.B.) #8

Hi Peter,

thanks for the suggestion. I also thought about this, but since I'm
currently sending too many different queryTypes around I didn't want
to do this right now; Its more that I currently try to find out what
exactly can later be put together in a seperate object holding all
things required for any query;

Regards

On 31 Mrz., 21:30, Karussell tableyourt...@googlemail.com wrote:

Hi,

there. I'm using a separate persistable object in my wicket app:

https://github.com/karussell/Jetwick/blob/master/src/main/java/de/jet...

or a bit easier:

https://github.com/karussell/ElasticSearchExample/blob/master/src/mai...

Regards,
Peter.


(system) #9