Just Pushed: Transport level compression, and recovery


(Shay Banon) #1

Hi,

Just pushed support for transport level compression. This can come very
handy in environments where the network is not the best one can setup out
there (did someone say certain cloud providers? :wink: ). Here is the issue:
http://github.com/elasticsearch/elasticsearch/issues/issue/321. By default,
it is not enabled.

On top of it being a setting, it can also be controlled on the (internal)
API level when one node communicated with another. A good place where it
makes sense to always use transport level compression is when one peer
recovers from another. This is what happens now (
http://github.com/elasticsearch/elasticsearch/issues/issue/322). From
initial tests, it seems like the index gets compressed between 40 - 60 %
when its transported over to the other node.

There are other areas where using compression make sense. For example,
storing the _source field compressed in the index. This is certainly
something I plan to tackle as well. (note, good compression support does not
end with compressed before write to "disk" and decompress after read from
"disk". in the _source example, it make sense to decompress only when really
needed, and even then in a chunked manner, so it will continue to "flow"
between nodes in its compressed form).

-shay.banon


(system) #2