Is it possible to send gzip compressed requests to ES?

C_Keser · February 23, 2013, 12:35pm

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite large
(a few megabytes), I want to compress the search request I'm sending to ES.
Trying this with curl has proved unsuccessful so far, using the command:

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data @req.txt.gz
http://localhost:9200/test1/_search?pretty=true

ES just replies with a generic search response (the one you get when you
send an empty post request to search url). If I remove the content encoding
header ES replies with a SearchParseException (as expected because it can't
parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clinton_Gormley · February 23, 2013, 1:49pm

On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite
large (a few megabytes), I want to compress the search request I'm
sending to ES. Trying this with curl has proved unsuccessful so far,
using the command:

You need to enable http compression on each node, by adding this to your
config file:

http.compression: true

See Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data
@req.txt.gz http://localhost:9200/test1/_search\?pretty\=true

ES just replies with a generic search response (the one you get when
you send an empty post request to search url). If I remove the content
encoding header ES replies with a SearchParseException (as expected
because it can't parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

C_Keser · February 23, 2013, 1:56pm

Already did that. Also doesn't setting http.compression affect the
responses (from ES) only? I want ES to accept compressed requests.

23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton Gormley yazdı:

On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite
large (a few megabytes), I want to compress the search request I'm
sending to ES. Trying this with curl has proved unsuccessful so far,
using the command:

You need to enable http compression on each node, by adding this to your
config file:
http.compression: true 
See Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data
@req.txt.gz http://localhost:9200/test1/_search\?pretty\=true

ES just replies with a generic search response (the one you get when
you send an empty post request to search url). If I remove the content
encoding header ES replies with a SearchParseException (as expected
because it can't parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vineeth_mohan · February 23, 2013, 2:14pm

On a different note , can i compress the response from ES ?

Thanks
Vineeth

On Sat, Feb 23, 2013 at 7:26 PM, C. Keser ckeser@gmail.com wrote:

Already did that. Also doesn't setting http.compression affect the
responses (from ES) only? I want ES to accept compressed requests.

23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton Gormley yazdı:
On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite
large (a few megabytes), I want to compress the search request I'm
sending to ES. Trying this with curl has proved unsuccessful so far,
using the command:

You need to enable http compression on each node, by adding this to your
config file:
http.compression: true
See Elasticsearch Platform — Find real-time answers at scale | Elastic http://www.elasticsearch.org/guide/reference/modules/http.html

clint

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data
@req.txt.gz http://localhost:9200/test1/_**search\?pretty\=true http://localhost:9200/test1/_search\?pretty\=true

ES just replies with a generic search response (the one you get when
you send an empty post request to search url). If I remove the content
encoding header ES replies with a SearchParseException (as expected
because it can't parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clinton_Gormley · February 23, 2013, 2:16pm

On Sat, 2013-02-23 at 05:56 -0800, C. Keser wrote:

Already did that. Also doesn't setting http.compression affect the
responses (from ES) only? I want ES to accept compressed requests.

Oh right - hmmm... interesting

No idea I'm afraid

23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton Gormley
yazdı:
On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:
> Hi,
>
>
> I have a visual query generator in my application from which
user can
> generate complex and large ES queries. As these queries may
be quite
> large (a few megabytes), I want to compress the search
request I'm
> sending to ES. Trying this with curl has proved unsuccessful
so far,
> using the command:
    You need to enable http compression on each node, by adding
    this to your 
    config file: 
    
        http.compression: true 
    
    See
    http://www.elasticsearch.org/guide/reference/modules/http.html 
    
    clint 
    
    
    > 
    > 
    > curl -v -i --compressed -H "Content-encoding: gzip" -X POST
    --data 
    > @req.txt.gz http://localhost:9200/test1/_search\?pretty
    \=true 
    > 
    > 
    > 
    > 
    > 
    > ES just replies with a generic search response (the one you
    get when 
    > you send an empty post request to search url). If I remove
    the content 
    > encoding header ES replies with a SearchParseException (as
    expected 
    > because it can't parse compressed bytes as JSON). 
    > 
    > 
    > So, am I missing something here or ES just does not support
    compressed 
    > requests? 
    > 
    > -- 
    > You received this message because you are subscribed to the
    Google 
    > Groups "elasticsearch" group. 
    > To unsubscribe from this group and stop receiving emails
    from it, send 
    > an email to elasticsearc...@googlegroups.com. 
    > For more options, visit
    https://groups.google.com/groups/opt_out. 
    >   
    >   
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clinton_Gormley · February 23, 2013, 2:18pm

On Sat, 2013-02-23 at 19:44 +0530, Vineeth Mohan wrote:

On a different note , can i compress the response from ES ?

yes - set http.compression: true, and include this header in your
request:

Accept-Encoding: gzip, deflate

clint

Thanks
Vineeth

On Sat, Feb 23, 2013 at 7:26 PM, C. Keser ckeser@gmail.com wrote:
Already did that. Also doesn't setting http.compression affect
the responses (from ES) only? I want ES to accept compressed
requests.

    23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton
    Gormley yazdı:
            On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote: 
            > Hi, 
            > 
            > 
            > I have a visual query generator in my application
            from which user can 
            > generate complex and large ES queries. As these
            queries may be quite 
            > large (a few megabytes), I want to compress the
            search request I'm 
            > sending to ES. Trying this with curl has proved
            unsuccessful so far, 
            > using the command: 
            
            You need to enable http compression on each node, by
            adding this to your 
            config file: 
            
                http.compression: true 
            
            See
            http://www.elasticsearch.org/guide/reference/modules/http.html 
            
            clint 
            
            
            > 
            > 
            > curl -v -i --compressed -H "Content-encoding: gzip"
            -X POST --data 
            > @req.txt.gz http://localhost:9200/test1/_search
            \?pretty\=true 
            > 
            > 
            > 
            > 
            > 
            > ES just replies with a generic search response (the
            one you get when 
            > you send an empty post request to search url). If I
            remove the content 
            > encoding header ES replies with a
            SearchParseException (as expected 
            > because it can't parse compressed bytes as JSON). 
            > 
            > 
            > So, am I missing something here or ES just does not
            support compressed 
            > requests? 
            > 
            > -- 
            > You received this message because you are subscribed
            to the Google 
            > Groups "elasticsearch" group. 
            > To unsubscribe from this group and stop receiving
            emails from it, send 
            
            > an email to elasticsearc...@googlegroups.com. 
            > For more options, visit
            https://groups.google.com/groups/opt_out. 
            >   
            >

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

simonw_2 · February 23, 2013, 8:57pm

On Saturday, February 23, 2013 2:56:26 PM UTC+1, C. Keser wrote:

Already did that. Also doesn't setting http.compression affect the
responses (from ES) only? I want ES to accept compressed requests.

this setting should trigger ES to accept gzip too. I think you need to
write "Content-Encoding" with Uppercase first letters can you try this:

curl -v -i --compressed -H "Content-Encoding: gzip" -X POST --data @req.txt.gz
http://localhost:9200/test1/_search\?pretty\=true

simon

23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton Gormley yazdı:
On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite
large (a few megabytes), I want to compress the search request I'm
sending to ES. Trying this with curl has proved unsuccessful so far,
using the command:

You need to enable http compression on each node, by adding this to your
config file:
http.compression: true 
See Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data
@req.txt.gz http://localhost:9200/test1/_search\?pretty\=true

ES just replies with a generic search response (the one you get when
you send an empty post request to search url). If I remove the content
encoding header ES replies with a SearchParseException (as expected
because it can't parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

C_Keser · February 23, 2013, 9:37pm

HTTP headers are case insensitive according to specs but i gave it a try
anyway; the response is still a generic search response...

I wonder if anyone else is able to reproduce this issue...?

23 Şubat 2013 Cumartesi 22:57:38 UTC+2 tarihinde simonw yazdı:

On Saturday, February 23, 2013 2:56:26 PM UTC+1, C. Keser wrote:

Already did that. Also doesn't setting http.compression affect the
responses (from ES) only? I want ES to accept compressed requests.

this setting should trigger ES to accept gzip too. I think you need to
write "Content-Encoding" with Uppercase first letters can you try this:

curl -v -i --compressed -H "Content-Encoding: gzip" -X POST --data @req.
txt.gz http://localhost:9200/test1/_search\?pretty\=true

simon
23 Şubat 2013 Cumartesi 15:49:45 UTC+2 tarihinde Clinton Gormley yazdı:
On Sat, 2013-02-23 at 04:35 -0800, C. Keser wrote:

Hi,

I have a visual query generator in my application from which user can
generate complex and large ES queries. As these queries may be quite
large (a few megabytes), I want to compress the search request I'm
sending to ES. Trying this with curl has proved unsuccessful so far,
using the command:

You need to enable http compression on each node, by adding this to your
config file:
http.compression: true 
See Elasticsearch Platform — Find real-time answers at scale | Elastic

clint

curl -v -i --compressed -H "Content-encoding: gzip" -X POST --data
@req.txt.gz http://localhost:9200/test1/_search\?pretty\=true

ES just replies with a generic search response (the one you get when
you send an empty post request to search url). If I remove the content
encoding header ES replies with a SearchParseException (as expected
because it can't parse compressed bytes as JSON).

So, am I missing something here or ES just does not support compressed
requests?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · February 24, 2013, 8:32am

Try something like this

curl -XPOST -H "Accept-Encoding: gzip" -H "Content-Encoding: gzip"
--data-binary @query.json.gz --compressed localhost:9200/_search

Jörg

Am 23.02.13 22:37, schrieb C. Keser:

HTTP headers are case insensitive according to specs but i gave it a
try anyway; the response is still a generic search response...

I wonder if anyone else is able to reproduce this issue...?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

C_Keser · February 24, 2013, 12:20pm

Thanks Jörg, "--data-binary" flag solved the problem!

24 Şubat 2013 Pazar 10:32:44 UTC+2 tarihinde Jörg Prante yazdı:

Try something like this

curl -XPOST -H "Accept-Encoding: gzip" -H "Content-Encoding: gzip"
--data-binary @query.json.gz --compressed localhost:9200/_search

Jörg

Am 23.02.13 22:37, schrieb C. Keser:

HTTP headers are case insensitive according to specs but i gave it a
try anyway; the response is still a generic search response...

I wonder if anyone else is able to reproduce this issue...?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
How to compress data when write into es Elasticsearch	5	2823	March 17, 2022
PUT gzipped data into elasticsearch Elasticsearch	9	4435	July 6, 2017
What kind of encoding does the ES support? Elasticsearch	4	20205	July 6, 2017
The effective way to update elasticsearch.yml Elasticsearch	6	3191	July 6, 2017
Disabling Elasticsearch Index compression version 7.10+ Elasticsearch	7	242	May 15, 2024

Is it possible to send gzip compressed requests to ES?

Related Topics