Using Java classes on the client instead of JSON


(Jacob Stultz) #1

I'm wondering how feasible it is to use some of the Java classes defined by
Elasticsearch on the client side, rather than building up JSON strings or
XContentBuilder instances manually. Specifically, I'm wondering about
creating mappings. It would be really nice if I could create a
DocumentMapper instance (which implements ToXContent) to define a mapping,
and then use that to create an XContentBuilder to pass to setSource on
PutMappingRequestBuilder.

This would be great, since it would rely on existing types to ensure that
valid mappings are created, rather than leaving open the possibility of
incorrectly constructing JSON (either via typos or incorrect field names,
or just incorrect structure).

However, I'm having trouble figuring out whether this is possible or
intended; it seems to create a DocumentMapper instance, I'd need a
DocumentMapper.Builder instance, which requires a DocumentMapperParser
instance to build the DocumentMapper, and the DocumentMapperParser instance
in turn requires a number of other objects not readily available.

Am I basically stuck with writing my own code to generate these JSON
structures? If so, it'd be really nice in the future if the Java API would
leverage the fact that much of the code is already used on the server side,
and make those classes easily available for clients to leverage static
typing to help ensure correctness.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #2

You should absolutely use the Java API and not use straight JSON if you are
using the Java client. The Java API is not well documented, but the best
way to see how it is utilized is by examining the test classes or the
corresponding RestAction class. In this case, RestPutMappingAction won't
help you since it uses JSON, but there are the test classes. Take a look at

https://github.com/elasticsearch/elasticsearch/tree/master/src/test/java/org/elasticsearch/indices/mapping

https://github.com/elasticsearch/elasticsearch/blob/master/src/test/java/org/elasticsearch/index/mapper/source/DefaultSourceMappingTests.java

I do not use the Java API for mappings since I prefer to use file-based
(dynamic templates) solutions since I can store the files in version
control, but the API is somewhere there.

--
Ivan

On Fri, Mar 28, 2014 at 2:20 PM, Jacob Stultz jacob@strava.com wrote:

I'm wondering how feasible it is to use some of the Java classes defined
by Elasticsearch on the client side, rather than building up JSON strings
or XContentBuilder instances manually. Specifically, I'm wondering about
creating mappings. It would be really nice if I could create a
DocumentMapper instance (which implements ToXContent) to define a mapping,
and then use that to create an XContentBuilder to pass to setSource on
PutMappingRequestBuilder.

This would be great, since it would rely on existing types to ensure that
valid mappings are created, rather than leaving open the possibility of
incorrectly constructing JSON (either via typos or incorrect field names,
or just incorrect structure).

However, I'm having trouble figuring out whether this is possible or
intended; it seems to create a DocumentMapper instance, I'd need a
DocumentMapper.Builder instance, which requires a DocumentMapperParser
instance to build the DocumentMapper, and the DocumentMapperParser instance
in turn requires a number of other objects not readily available.

Am I basically stuck with writing my own code to generate these JSON
structures? If so, it'd be really nice in the future if the Java API would
leverage the fact that much of the code is already used on the server side,
and make those classes easily available for clients to leverage static
typing to help ensure correctness.

Thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAp32JA6MdG9KYicGLvaUXSb9VwH3ed%2Bf_sAqxfzAR-aQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #3

The DocumentMapper is located at server side or in a plugin when the
request of a client is processed. It is not possible to use it by clients
to bypass the ES API. The communication to the nodes that receive document
mapper requests is very low level and beyond the ES API - see the
transport shard actions.

In ES, the XContent universal helper classes wrap all possible formats,
like SMILE or YAML, on the ES API and for internal transmissions too.
Compressed JSON is used for data transmission. Unless exceptions, no Java
object serialization is used.

In my eyes it is one of the strengths of XContent to construct arbitrary
JSON-like object/array streams without a schema. You don't have to care
about JSON strings at all. It is a natural fluent API that helps a lot in
writing correct code (if your IDE uses code completion).

The alternative would be a static approach where every action class would
carry its own parameterization and serialization scheme. This is one area
where ES really shines - it has a dynamic parameterization, XContent can
drive new actions and existent actions as well. The price is missing
validation at code writing time, but I don't miss such a thing.

Jörg

On Fri, Mar 28, 2014 at 10:20 PM, Jacob Stultz jacob@strava.com wrote:

I'm wondering how feasible it is to use some of the Java classes defined
by Elasticsearch on the client side, rather than building up JSON strings
or XContentBuilder instances manually. Specifically, I'm wondering about
creating mappings. It would be really nice if I could create a
DocumentMapper instance (which implements ToXContent) to define a mapping,
and then use that to create an XContentBuilder to pass to setSource on
PutMappingRequestBuilder.

This would be great, since it would rely on existing types to ensure that
valid mappings are created, rather than leaving open the possibility of
incorrectly constructing JSON (either via typos or incorrect field names,
or just incorrect structure).

However, I'm having trouble figuring out whether this is possible or
intended; it seems to create a DocumentMapper instance, I'd need a
DocumentMapper.Builder instance, which requires a DocumentMapperParser
instance to build the DocumentMapper, and the DocumentMapperParser instance
in turn requires a number of other objects not readily available.

Am I basically stuck with writing my own code to generate these JSON
structures? If so, it'd be really nice in the future if the Java API would
leverage the fact that much of the code is already used on the server side,
and make those classes easily available for clients to leverage static
typing to help ensure correctness.

Thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFD%3DoSJTL2Uzw%2B5Acwm9Lrdk6VueM-%2BRJj%3DPu2zX9zH0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jacob Stultz) #4

Thanks, appreciate the quick response and I'll just move forward with using
XContentBuilder then.

I do want to be clear that my intent was not to "bypass" the ES API, but
rather I was thinking that it would be very useful for things like
mappings, where the structure and content is well defined by ElasticSearch
and there are rules determining whether a mapping is valid or invalid
beyond just JSON syntax (unlike documents, for example), to have a stronger
interface for creating them on the client side, even if it would just get
serialized into the same JSON for transport to the server. Given that there
are already classes written for the server that represent mappings and
fields, and are readily serializable to JSON, giving JVM clients the
ability to leverage these would be very useful.

One followup question, regarding XContentBuilder. I'd like to be able to
define some simple helper classes for defining field mappings with common
configurations within my project. It's not clear how to cleanly delegate
construction of substructures to other methods or classes. I would love to
be able to do something like the following (in Scala)

XContentFactory.jsonBuilder()
.startObject
.startObject("properties")
.field("subject", stringField(store = true))
.field("content", stringField())
.endObject
.endObject

where stringField is something that I implement. It would be ideal if that
just returned an XContentBuilder as well, and there were a version of the
field method that would accept that (in addition to the ones that accept a
String, Double, Map, etc). However, it doesn't appear that it does, and I'm
at a loss to figure out a way to use XContentBuilder to build smaller
pieces that are then later composed into a larger whole. Is the only option
to either generate Map objects for the substructures, or rely on mutation
along these lines:

val mapping = XContentFactory.jsonBuilder()
.startObject
.startObject("properties")

addStringField(mapping, "subject", store = true)
addStringField(mapping, "content)

mapping.endObject.endObject

On Fri, Mar 28, 2014 at 3:45 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The DocumentMapper is located at server side or in a plugin when the
request of a client is processed. It is not possible to use it by clients
to bypass the ES API. The communication to the nodes that receive document
mapper requests is very low level and beyond the ES API - see the
transport shard actions.

In ES, the XContent universal helper classes wrap all possible formats,
like SMILE or YAML, on the ES API and for internal transmissions too.
Compressed JSON is used for data transmission. Unless exceptions, no Java
object serialization is used.

In my eyes it is one of the strengths of XContent to construct arbitrary
JSON-like object/array streams without a schema. You don't have to care
about JSON strings at all. It is a natural fluent API that helps a lot in
writing correct code (if your IDE uses code completion).

The alternative would be a static approach where every action class would
carry its own parameterization and serialization scheme. This is one area
where ES really shines - it has a dynamic parameterization, XContent can
drive new actions and existent actions as well. The price is missing
validation at code writing time, but I don't miss such a thing.

Jörg

On Fri, Mar 28, 2014 at 10:20 PM, Jacob Stultz jacob@strava.com wrote:

I'm wondering how feasible it is to use some of the Java classes defined
by Elasticsearch on the client side, rather than building up JSON strings
or XContentBuilder instances manually. Specifically, I'm wondering about
creating mappings. It would be really nice if I could create a
DocumentMapper instance (which implements ToXContent) to define a mapping,
and then use that to create an XContentBuilder to pass to setSource on
PutMappingRequestBuilder.

This would be great, since it would rely on existing types to ensure that
valid mappings are created, rather than leaving open the possibility of
incorrectly constructing JSON (either via typos or incorrect field names,
or just incorrect structure).

However, I'm having trouble figuring out whether this is possible or
intended; it seems to create a DocumentMapper instance, I'd need a
DocumentMapper.Builder instance, which requires a DocumentMapperParser
instance to build the DocumentMapper, and the DocumentMapperParser instance
in turn requires a number of other objects not readily available.

Am I basically stuck with writing my own code to generate these JSON
structures? If so, it'd be really nice in the future if the Java API would
leverage the fact that much of the code is already used on the server side,
and make those classes easily available for clients to leverage static
typing to help ensure correctness.

Thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/k-PoX-iCIeg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFD%3DoSJTL2Uzw%2B5Acwm9Lrdk6VueM-%2BRJj%3DPu2zX9zH0w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoFD%3DoSJTL2Uzw%2B5Acwm9Lrdk6VueM-%2BRJj%3DPu2zX9zH0w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP3WdZGCzsFxRvnZoe9Q2451%2BdAJD-aJvzVoARs%2BtuhgAh5x%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


#5

It is indeed possible to compose complex XContent by implementing helper methods and delegating construction to them as you propose. Indeed this is a common pattern within the Elasticsearch code itself.

The basic idea is that you pass your XContentBuilder to your helper methods which return a reference to the same builder after mutating it in some fashion. Just design your helper classes to have methods with this signature: @Override public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException.

Then you can simply delegate to your helper classes by passing your builder reference down to their toXContent() methods. For a real-world example see: https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/action/admin/indices/recovery/RecoveryResponse.java#L91

Hope that helps.

On Mar 28, 2014, at 4:32 PM, Jacob Stultz jacob@strava.com wrote:

Thanks, appreciate the quick response and I'll just move forward with using XContentBuilder then.

I do want to be clear that my intent was not to "bypass" the ES API, but rather I was thinking that it would be very useful for things like mappings, where the structure and content is well defined by ElasticSearch and there are rules determining whether a mapping is valid or invalid beyond just JSON syntax (unlike documents, for example), to have a stronger interface for creating them on the client side, even if it would just get serialized into the same JSON for transport to the server. Given that there are already classes written for the server that represent mappings and fields, and are readily serializable to JSON, giving JVM clients the ability to leverage these would be very useful.

One followup question, regarding XContentBuilder. I'd like to be able to define some simple helper classes for defining field mappings with common configurations within my project. It's not clear how to cleanly delegate construction of substructures to other methods or classes. I would love to be able to do something like the following (in Scala)

XContentFactory.jsonBuilder()
.startObject
.startObject("properties")
.field("subject", stringField(store = true))
.field("content", stringField())
.endObject
.endObject

where stringField is something that I implement. It would be ideal if that just returned an XContentBuilder as well, and there were a version of the field method that would accept that (in addition to the ones that accept a String, Double, Map, etc). However, it doesn't appear that it does, and I'm at a loss to figure out a way to use XContentBuilder to build smaller pieces that are then later composed into a larger whole. Is the only option to either generate Map objects for the substructures, or rely on mutation along these lines:

val mapping = XContentFactory.jsonBuilder()
.startObject
.startObject("properties")

addStringField(mapping, "subject", store = true)
addStringField(mapping, "content)

mapping.endObject.endObject

On Fri, Mar 28, 2014 at 3:45 PM, joergprante@gmail.com joergprante@gmail.com wrote:
The DocumentMapper is located at server side or in a plugin when the request of a client is processed. It is not possible to use it by clients to bypass the ES API. The communication to the nodes that receive document mapper requests is very low level and beyond the ES API - see the transport shard actions.

In ES, the XContent universal helper classes wrap all possible formats, like SMILE or YAML, on the ES API and for internal transmissions too. Compressed JSON is used for data transmission. Unless exceptions, no Java object serialization is used.

In my eyes it is one of the strengths of XContent to construct arbitrary JSON-like object/array streams without a schema. You don't have to care about JSON strings at all. It is a natural fluent API that helps a lot in writing correct code (if your IDE uses code completion).

The alternative would be a static approach where every action class would carry its own parameterization and serialization scheme. This is one area where ES really shines - it has a dynamic parameterization, XContent can drive new actions and existent actions as well. The price is missing validation at code writing time, but I don't miss such a thing.

Jörg

On Fri, Mar 28, 2014 at 10:20 PM, Jacob Stultz jacob@strava.com wrote:
I'm wondering how feasible it is to use some of the Java classes defined by Elasticsearch on the client side, rather than building up JSON strings or XContentBuilder instances manually. Specifically, I'm wondering about creating mappings. It would be really nice if I could create a DocumentMapper instance (which implements ToXContent) to define a mapping, and then use that to create an XContentBuilder to pass to setSource on PutMappingRequestBuilder.

This would be great, since it would rely on existing types to ensure that valid mappings are created, rather than leaving open the possibility of incorrectly constructing JSON (either via typos or incorrect field names, or just incorrect structure).

However, I'm having trouble figuring out whether this is possible or intended; it seems to create a DocumentMapper instance, I'd need a DocumentMapper.Builder instance, which requires a DocumentMapperParser instance to build the DocumentMapper, and the DocumentMapperParser instance in turn requires a number of other objects not readily available.

Am I basically stuck with writing my own code to generate these JSON structures? If so, it'd be really nice in the future if the Java API would leverage the fact that much of the code is already used on the server side, and make those classes easily available for clients to leverage static typing to help ensure correctness.

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8cbf0f15-a5d2-4dda-9759-ede3505ca08e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/k-PoX-iCIeg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFD%3DoSJTL2Uzw%2B5Acwm9Lrdk6VueM-%2BRJj%3DPu2zX9zH0w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAP3WdZGCzsFxRvnZoe9Q2451%2BdAJD-aJvzVoARs%2BtuhgAh5x%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E0A23F5C-3DCA-4E71-8878-F8EF4EBECE85%40elasticsearch.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6