ElasticSearch and Wicket -> serialization nightmare

K_B · March 30, 2011, 2:20pm

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

kimchy · March 30, 2011, 2:46pm

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...
On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

K_B · March 30, 2011, 5:45pm

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

kimchy · March 30, 2011, 6:03pm

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).
On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

K_B · March 30, 2011, 6:18pm

I currently pass a SearchSourceBilder around and manipulate it at
certain places. However I need to be able to persist the result of the
manipulations inside a class that wants it content to be serializable
(no further manipulation afterwards beside paging);
Now, the bye from sourceBuilder is the whole query, right? Is there
any way to send this to ES and only manipulate
the .setFrom(int).SetSize(int) at it?

Or what else could be passed around to later recreate a new
SearchRequestBuilder based upon it (I saw that SearchRequestBuilder
accepts Query, Facet and Filter as byte)

Thanks for keeping up with me

On 30 Mrz., 20:03, Shay Banon shay.ba...@elasticsearch.com wrote:

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).

On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

K_B · March 31, 2011, 10:18am

In case other have similar problems:

Solution is to build the SearchSourceBuilder as bytes;

one so can do:

this.sourceBuilder = sourceBuilder.buildAsBytes(XContentType.SMILE);

and have

private byte sourceBuilder;

on class level;

Search then can be used like:

client().search(new SearchRequest(INDEX, byte
sourceBuilder).extraSource(any extra byte extraSource)

so a precompiled search can be narrowed or limited to certain results;

e.g:

getClient().search(new
SearchRequest(searchindex).source(source).extraSource(extraSource)).actionGet()

this now allows fast and very nice queries against any constructed
query as well as storing for later use; One so can build DataProviders
for wicket, historize search-queries for easy use in search
breadcrumbing, complete transparent backbutton support, etc.;
Performance didn't seem to get any big impact;

Any comments appreciated;

On 30 Mrz., 20:18, "K.B." korbinian.ba...@googlemail.com wrote:

I currently pass a SearchSourceBilder around and manipulate it at
certain places. However I need to be able to persist the result of the
manipulations inside a class that wants it content to be serializable
(no further manipulation afterwards beside paging);
Now, the bye from sourceBuilder is the whole query, right? Is there
any way to send this to ES and only manipulate
the .setFrom(int).SetSize(int) at it?

Or what else could be passed around to later recreate a new
SearchRequestBuilder based upon it (I saw that SearchRequestBuilder
accepts Query, Facet and Filter as byte)

Thanks for keeping up with me

On 30 Mrz., 20:03, Shay Banon shay.ba...@elasticsearch.com wrote:

You can't build back from json a SearchSourceBuilder, its not really needed in the flow of usage (requires a whole parsing layer to build it back).

On Wednesday, March 30, 2011 at 7:45 PM, K.B. wrote:

Hi,

I don't want to argue now why it makes sense vs. it might be bad -
there are enough reasons why both things are ok;

Simply said, what I need is a way to make a SearchSourceBuilder
"saveable"; It has a nice byte buildAsBytes() and toXContent(...)
function, but I can't find a way to construct a new
SearchSourceBuilder out of this; I looked at the forums plus I tried
things like .query(byte) etc. but haven't had any success;

Best

On 30 Mrz., 16:46, Shay Banon shay.ba...@elasticsearch.com wrote:

I don't really understand why one would want to serialize the request or response, they are transient objects that are part of a "web" request.

There is an option to pass a json string as the search request. And, the response can be serialized to json (we've been over it in several other threads).

And no, elasticsearch is not going to add Serializable to add the request response objects. It has its own serialization mechanism, and its not worth it to add another serialization (Java one), especially since its bad, bloated, and slow.

Going back to the first point, it sounds really strange that those objects would need to be serialized, its costly...

On Wednesday, March 30, 2011 at 4:20 PM, K.B. wrote:

Hello,

I use ES together with wicket and all works fine but for one thing I
can't tame: Serialization

Wicket tries to keep state on the server side by serilizing the whole
page, however, as ES has not a single Serilizable Object in its java-
api I'm struggling all the time to overcome that problem. I actually
can easily stuff SearchHitField's content into a map of <String,
String> or do similr things with Facet etc. but I have no clue how to
solve the problem with SearchSourceBuilder that I pass around so
different panels can manipulate it.

So I really need some help how to solve this. Beside that, why is
there nearly no consumable, serializable interface in ES? At least,
all comes down to JSOn and this is just simple CHAR's and Strings
meaning serializing should be as easy as to put "implements
serializable" to the basic query interfaces.

Best

Karussell1 · March 31, 2011, 7:30pm

Hi,

there. I'm using a separate persistable object in my wicket app:

github.com

karussell/Jetwick/blob/master/src/main/java/de/jetwick/es/TweetQuery.java

/**
 * Copyright (C) 2010 Peter Karich <jetwick_@_pannous_._info>
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *         http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package de.jetwick.es;

import org.elasticsearch.index.query.FilterBuilder;
import org.elasticsearch.index.query.QueryBuilder;
import org.elasticsearch.index.query.FilterBuilders;

This file has been truncated. show original

or a bit easier:

github.com

karussell/ElasticSearchExample/blob/master/src/main/java/de/jetwick/ese/search/MyQuery.java

/*
 * Copyright 2011 Peter Karich, peat_hal 'at' users 'dot' sourceforge 'dot' net.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package de.jetwick.ese.search;

import org.elasticsearch.index.query.xcontent.FilterBuilders;
import org.elasticsearch.index.query.xcontent.XContentFilterBuilder;
import java.util.Set;

This file has been truncated. show original

Regards,
Peter.

K_B · April 1, 2011, 2:16pm

Hi Peter,

thanks for the suggestion. I also thought about this, but since I'm
currently sending too many different queryTypes around I didn't want
to do this right now; Its more that I currently try to find out what
exactly can later be put together in a seperate object holding all
things required for any query;

Regards

On 31 Mrz., 21:30, Karussell tableyourt...@googlemail.com wrote:

Hi,

there. I'm using a separate persistable object in my wicket app:

https://github.com/karussell/Jetwick/blob/master/src/main/java/de/jet...

or a bit easier:

https://github.com/karussell/ElasticSearchExample/blob/master/src/mai...

Regards,
Peter.