How to compress data when write into es

Elastic Stack Elasticsearch
1

Hi team,

I want to build an ES cluster for services in other regions, so ES needs to be exposed to the public network.

The amount of data transferred is 2 T /day , it's huge .
So I expect to compress before writing into ES to save bandwidth transfer costs.

Is there any good way to achieve this or it has to be done through a new component?
Any suggestions would be very helpful to me, thanks!

2

It depends entirely on how are you planning to send the data as the compression should be done before sending the data to Elasticsearch.

If you are going to use Logstash or Filebeat then there is no issue, both tools support compression with the Elasticsearch output, it is not enabled per default but is pretty easy to configure.

If you are going to use other tool or a custom application, then you will need to check how to send a compressed http request to Elasticsearch on each case.

3 Likes
3

Thanks @leandrojmp !

Follow your advice，I found this http_compression parameter in logstash.

Does it mean that logstash can automatically send data to Elasticsearch in compressed format after it is turned on?

And Elasticsearch can automatically handle writing in compressed format?

4

Yes, if you set this option to true, logstash will send compressed data to Elasticsearch and Elasticsearch will decompress the data when it received the requests, no other change is needed.

5

Thanks a lot , it's worked !

echo '{  "@timestamp": "2099-11-16T13:12:00",  "message": "GET /search HTTP/1.1 200 1070000",  "user": {    "id": "hello"  }}'  \
|gzip \
|curl -v -i --data-binary @- -H "Content-Encoding: gzip" -H 'Content-Type: application/json' -u "elastic:password" 'http://1.2.3.4:9200/my-index-000001/_doc/?pretty'
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.