PUT gzipped data into elasticsearch

Marcel_Matus · March 24, 2015, 10:59am

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over the
network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 24, 2015, 12:50pm

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over the
network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marcel_Matus · March 24, 2015, 1:30pm

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the complete
way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over
the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 24, 2015, 9:18pm

Logstash has both Java and HTTP output, but I assume you want to use HTTP.

Set http.compression parameter to true in ES configuration, then you can
use gzip-compressed HTTP traffic using "Accept-Encoding" header.

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the complete
way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over
the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marcel_Matus · March 25, 2015, 7:03am

Actually, I am more interested about pushing compressed data into
elasticsearch than asking for compressed data.
So far, I tried it manually, using Google chrome plugin - postman, or curl,
without success:

curl -sS -X POST --data-binary txn001.json.gz http://localhost:9200/gzip/new
-H "Content-Type: application/json" -H "Content-Encoding: gzip"
Response: curl: (52) Empty reply from server

Marcel

On Tue, Mar 24, 2015 at 10:18 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Logstash has both Java and HTTP output, but I assume you want to use HTTP.

Set http.compression parameter to true in ES configuration, then you can
use gzip-compressed HTTP traffic using "Accept-Encoding" header.

HTTP | Elasticsearch Guide [8.11] | Elastic

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the
complete way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over
the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Karl_Putland · March 25, 2015, 12:41pm

I would guess that your gzipped data would need to be base64 encoded and
placed in a field of the JSON along with an extract of whatever other
fields you find important or interesting. Then you'll have _source.gzipped
available to your application to decode and unzip when you choose.

--Karl
On Mar 25, 2015 1:04 AM, "Marcel Matus" matusmarcel@gmail.com wrote:

Actually, I am more interested about pushing compressed data into
elasticsearch than asking for compressed data.
So far, I tried it manually, using Google chrome plugin - postman, or
curl, without success:

curl -sS -X POST --data-binary txn001.json.gz
http://localhost:9200/gzip/new -H "Content-Type: application/json" -H
"Content-Encoding: gzip"
Response: curl: (52) Empty reply from server

Marcel

On Tue, Mar 24, 2015 at 10:18 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Logstash has both Java and HTTP output, but I assume you want to use HTTP.

Set http.compression parameter to true in ES configuration, then you can
use gzip-compressed HTTP traffic using "Accept-Encoding" header.

HTTP | Elasticsearch Guide [8.11] | Elastic

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the
complete way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over
the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CA%2BEXWsz%2Bhg_P2E04_ZJOZmszMhNMZhDf9z3R57HJf46uOJjJHQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 25, 2015, 1:15pm

Yes, I mean pushing compressed data.

You have several wrong assumptions: use PUT instead of POST, mark file with
a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and use
URL for a document creation 'localhost:9200/test/docs/1' for example.

Here is my full demo of a correct curl command

gist.github.com

https://gist.github.com/jprante/15f570be70504285a678

binary-put-es.sh

# first, set
#
# http.compression: true
#
# in config/elasticsearch.yml

curl -XDELETE 'localhost:9200/test'

rm -f content.json content.json.gz
echo '{"content":"Hello World"}' > content.json

This file has been truncated. show original

Jörg

On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmarcel@gmail.com wrote:

Actually, I am more interested about pushing compressed data into
elasticsearch than asking for compressed data.
So far, I tried it manually, using Google chrome plugin - postman, or
curl, without success:

curl -sS -X POST --data-binary txn001.json.gz
http://localhost:9200/gzip/new -H "Content-Type: application/json" -H
"Content-Encoding: gzip"
Response: curl: (52) Empty reply from server

Marcel

On Tue, Mar 24, 2015 at 10:18 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Logstash has both Java and HTTP output, but I assume you want to use HTTP.

Set http.compression parameter to true in ES configuration, then you can
use gzip-compressed HTTP traffic using "Accept-Encoding" header.

HTTP | Elasticsearch Guide [8.11] | Elastic

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the
complete way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions of
those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send over
the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEiKwvCWtRrLZ1OrorZp-oJPpQvqbBNAmOJPA-%3D%2Bba5zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marcel_Matus · March 25, 2015, 8:02pm

Hi Jorg,
thank you for your help - mainly your practical example is great. The
header Accept-Encoding isn't really necessary. I was almost there, but was
missing either the id of document or before the binary file.

Thanks.
Marcel

On Wed, Mar 25, 2015 at 2:15 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Yes, I mean pushing compressed data.

You have several wrong assumptions: use PUT instead of POST, mark file
with a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and
use URL for a document creation 'localhost:9200/test/docs/1' for example.

Here is my full demo of a correct curl command

Put binary data to Elasticsearch · GitHub

Jörg

On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Actually, I am more interested about pushing compressed data into
elasticsearch than asking for compressed data.
So far, I tried it manually, using Google chrome plugin - postman, or
curl, without success:

curl -sS -X POST --data-binary txn001.json.gz
http://localhost:9200/gzip/new -H "Content-Type: application/json" -H
"Content-Encoding: gzip"
Response: curl: (52) Empty reply from server

Marcel

On Tue, Mar 24, 2015 at 10:18 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Logstash has both Java and HTTP output, but I assume you want to use
HTTP.

Set http.compression parameter to true in ES configuration, then you can
use gzip-compressed HTTP traffic using "Accept-Encoding" header.

HTTP | Elasticsearch Guide [8.11] | Elastic

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the
complete way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to process
these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions
of those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send
over the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEiKwvCWtRrLZ1OrorZp-oJPpQvqbBNAmOJPA-%3D%2Bba5zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEiKwvCWtRrLZ1OrorZp-oJPpQvqbBNAmOJPA-%3D%2Bba5zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHUcNyg6iuaNa88D-koRb9j37nYvKh_8j29cSUt_%2ByMViz%2Brbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

jprante · March 25, 2015, 8:29pm

Oh sorry, I forgot the curl parameter --compressed

Now added to the gist.

Jörg

On Wed, Mar 25, 2015 at 9:02 PM, Marcel Matus matusmarcel@gmail.com wrote:

Hi Jorg,
thank you for your help - mainly your practical example is great. The
header Accept-Encoding isn't really necessary. I was almost there, but was
missing either the id of document or before the binary file.

Thanks.
Marcel

On Wed, Mar 25, 2015 at 2:15 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Yes, I mean pushing compressed data.

You have several wrong assumptions: use PUT instead of POST, mark file
with a preceding '@', set 'Accept-Encoding' header to 'gzip,deflate', and
use URL for a document creation 'localhost:9200/test/docs/1' for example.

Here is my full demo of a correct curl command

Put binary data to Elasticsearch · GitHub

Jörg

On Wed, Mar 25, 2015 at 8:03 AM, Marcel Matus matusmarcel@gmail.com
wrote:

Actually, I am more interested about pushing compressed data into
elasticsearch than asking for compressed data.
So far, I tried it manually, using Google chrome plugin - postman, or
curl, without success:

curl -sS -X POST --data-binary txn001.json.gz
http://localhost:9200/gzip/new -H "Content-Type: application/json" -H
"Content-Encoding: gzip"
Response: curl: (52) Empty reply from server

Marcel

On Tue, Mar 24, 2015 at 10:18 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Logstash has both Java and HTTP output, but I assume you want to use
HTTP.

Set http.compression parameter to true in ES configuration, then you
can use gzip-compressed HTTP traffic using "Accept-Encoding" header.

HTTP | Elasticsearch Guide [8.11] | Elastic

Jörg

On Tue, Mar 24, 2015 at 2:30 PM, Marcel Matus matusmarcel@gmail.com
wrote:

Hi Joe,
no, we post data into elasticsearch using Logstash. Actually, the
complete way is: APP - Logstash - RMQ - Logstash - ES.
And when I include F5 VIP's, there are 6 services on the way to
process these data...

Marcel

On Tue, Mar 24, 2015 at 1:50 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Are you using Java API?

Jörg

On Tue, Mar 24, 2015 at 11:59 AM, Marcel Matus <matusmarcel@gmail.com

wrote:

Hi,
some of our data are big ones (1 - 10 MB), and if there are milions
of those, it causes us trouble in our internal network.
We would like to compress these data in generation time, and send
over the network to elasticsearch in compressed state. I tried to find some
solution, but usually I found only retrieving gzipped data. Is there a way,
how to put compressed data into elasticsearch?

Thanks,
Marcel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bce57798-0b51-4c9e-a0ca-06e41b2ca160%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFagGZYQ1wCZe%2BCMrUSUJ9gE7erjA7fk1aSOLfsdZkXjg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyj4cAi0oDGqgLmeSxLwVwtCssF72pyhufKV8ML911auHA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG%3DzuBnV4LWr7ySj6arirFzjxgCARzq%2Bvh05mFqyJ7%2Bpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyhFUHHgwQwubaqPed0O6oWrHJaeD_nousLF%3DJW69p_1zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/iWT19g3_Viw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEiKwvCWtRrLZ1OrorZp-oJPpQvqbBNAmOJPA-%3D%2Bba5zg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEiKwvCWtRrLZ1OrorZp-oJPpQvqbBNAmOJPA-%3D%2Bba5zg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Marcel Matus

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyg6iuaNa88D-koRb9j37nYvKh_8j29cSUt_%2ByMViz%2Brbg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHUcNyg6iuaNa88D-koRb9j37nYvKh_8j29cSUt_%2ByMViz%2Brbg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUEkTprkFb0ieNE0Q807rnHwXU7ukroGwW_9gr0BuE5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Is it possible to send gzip compressed requests to ES? Elasticsearch	10	16919	July 6, 2017
Bulk API does not work for huge data! Elasticsearch	4	1144	July 6, 2017
Compressing http responses Elasticsearch	5	2885	July 6, 2017
How to compress data when write into es Elasticsearch	5	2893	March 17, 2022
Http.compression Elasticsearch	2	472	July 6, 2017

PUT gzipped data into elasticsearch

Related topics