Convert bulk request to json document and publish that document to ES as a seperate task

Hi,

I'm expecting to implement ES back statistic collecting application. For
that I'm expecting to publish data using bulk API. I want to separate out
request generating and request publishing tasks. First task is creating
bulk request using JAVA API and write that request to a temp file as a JSON
document and then request publisher publishes the requests which are in the
files.
I have tried to write created bulk request using writeTo method as follows.
It writes the request as a semi-json document containing some random weird
bytes(resulting file is attached).

        FileOutputStream fileOutputStream = new FileOutputStream(new 

File("request.txt"));
OutputStreamStreamOutput streamOutput = new
OutputStreamStreamOutput(fileOutputStream);
bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand
what goes wrong while writing the request and is it possible to implement
my application by separating request generating and publishing tasks as
mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You are using the binary stream protocol of ES in the writeTo() method
which is not appropriate for writing to files.

Once you added requests to a bulk request, you can not get your content
back as JSON.

A better approach is to use an XContentBuilder with an OutputStream, and
add the content to it, independent of BulkRequestBuilder.

Jörg

On Tue, Apr 28, 2015 at 8:21 PM, Manjula Piyumal manjulapiyumal@gmail.com
wrote:

Hi,

I'm expecting to implement ES back statistic collecting application. For
that I'm expecting to publish data using bulk API. I want to separate out
request generating and request publishing tasks. First task is creating
bulk request using JAVA API and write that request to a temp file as a JSON
document and then request publisher publishes the requests which are in the
files.
I have tried to write created bulk request using writeTo method as
follows. It writes the request as a semi-json document containing some
random weird bytes(resulting file is attached).

        FileOutputStream fileOutputStream = new FileOutputStream(new

File("request.txt"));
OutputStreamStreamOutput streamOutput = new
OutputStreamStreamOutput(fileOutputStream);
bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand
what goes wrong while writing the request and is it possible to implement
my application by separating request generating and publishing tasks as
mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGAVSjOWX6%2BQ08pNUBWczf6ySYwGDm%2BXMsepMnsWysvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Hi Jörg,

Thanks for the quick response. I'll try it in that way.

Thanks
Manjula

On Wed, Apr 29, 2015 at 12:32 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You are using the binary stream protocol of ES in the writeTo() method
which is not appropriate for writing to files.

Once you added requests to a bulk request, you can not get your content
back as JSON.

A better approach is to use an XContentBuilder with an OutputStream, and
add the content to it, independent of BulkRequestBuilder.

Jörg

On Tue, Apr 28, 2015 at 8:21 PM, Manjula Piyumal <manjulapiyumal@gmail.com

wrote:

Hi,

I'm expecting to implement ES back statistic collecting application. For
that I'm expecting to publish data using bulk API. I want to separate out
request generating and request publishing tasks. First task is creating
bulk request using JAVA API and write that request to a temp file as a JSON
document and then request publisher publishes the requests which are in the
files.
I have tried to write created bulk request using writeTo method as
follows. It writes the request as a semi-json document containing some
random weird bytes(resulting file is attached).

        FileOutputStream fileOutputStream = new FileOutputStream(new

File("request.txt"));
OutputStreamStreamOutput streamOutput = new
OutputStreamStreamOutput(fileOutputStream);
bulkRequestBuilder.request().writeTo(streamOutput);

I'm not sure whether I have missed something here. I want to understand
what goes wrong while writing the request and is it possible to implement
my application by separating request generating and publishing tasks as
mentioned above. Any help would be appreciated.

Thanks
Manjula

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/690f5a0a-3c62-43f3-b662-259b4dce79e1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/RCz0JgIP5kY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGAVSjOWX6%2BQ08pNUBWczf6ySYwGDm%2BXMsepMnsWysvA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGAVSjOWX6%2BQ08pNUBWczf6ySYwGDm%2BXMsepMnsWysvA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Manjula Piyumal De Silva

Software Engineer,
AdroitLogic Private Ltd,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFLAp0XqWcdKdnyrw3vg6%2Be4tCp%3DQY2MAYDrngKfEOMTkbpkUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.