How to implement deflate (best compression)?

Elastic Stack Elasticsearch
1

Hello, kindly help me out.
My elastic version is 6.3 how can i implement best compression for the log stored by elastic.
Thanks

2

Welcome to our community! :smiley:
Please upgrade, 6.3 is EOL and no longer supported. 7.15 is latest.

Take a look at Force merge | Elasticsearch Guide [7.15] | Elastic. You will need to apply the setting and then run a force merge.

3

Thanks for reply
Sorry for the wrong version i told you about elastic.
I am using graylog 3.0.2 and elastic 6.8.20.
Can i implement best compression without upgrade to 7.15?
And clearly tell me what are the various ways to apply best compression or there is only force merge can be implemented.
Thank in advance.

4

I am not sure about your version but I have done it for the 7.6 version like this :

PUT _template/my_template
{
  "index_patterns": ["*"],
  "settings": {
    "index": {
      "codec": "best_compression"
    }
  },
   "mappings" : {
     "_source": {
     "excludes": [
      ]
     }
     }
}

After this, all your new indices will get created with BEST compression enabled.

Note: Please try this in your lower environment first before you do it in a production cluster.
Also, the BEST compression is recommended only for logging/monitoring use cases and should not be implemented for catalog searches without doing performance benchmarking for your use case.

5

Thanks for the reply
But one thing i want to know how this command works what i need to do because i am unable to run it directly on terminal.
I need to write a file or else?
Or tell me how you applied it?
Kindly tell me.

6

don't you have kibana?

from there you can run the PUT api call directly from Dev tool.

If not then run curl command like :

curl -XPUT "http://<your cluster ip or fqdn>:9200/_template/my_template" -H 'Content-Type: application/json' -d'{ "index_patterns": ["*"], "settings": { "index": { "codec": "best_compression" } }, "mappings" : { "_source": { "excludes": [ ] } }}'

once you run the above command all your new indices will have best compression.

If you want to do it for the existing index then you will need to close it first and then change compression setting and then open the index.

something like this:

  1. Close all indices:http://localhost:9200/_all/_close'
  2. Apply best_compression to all curl -XPUT 'http://localhost:9200/_all/_settings' -d '{"index.codec" : "best_compression"}'
  3. Open all indices: curl -XPOST 'http://localhost:9200/_all/_open'
7

image
image1105×56 5.63 KB

Kindly guide it shows this error.

8

Is your ES running on port 9200? probably you changed the port in your configuration file?

Can you share the cluster stats or any other command output here so that we know it's connecting to the host and port correctly?

9

It is running on 9200.

image
image1063×64 2.7 KB

10

can you try this and attach the output, if this works then XPUT should also work.

curl -XGET "http://192.168.1.211:9200/_cluster/health"

11

image
image1274×121 7.41 KB

Getting this when using localhost on the place of ip.

12

its just an example , you can create template the way you want:
try this
curl -XPUT "http://localhost:9200/_template/my_template" -H 'Content-Type: application/json' -d'{ "index_patterns": ["*"], "settings": { "index": { "codec": "best_compression" } }}'

read more here: Create or update index template API | Elasticsearch Guide [7.15] | Elastic

13

Thanks a lot
How can i verify that the best compression is implemented after running the command.

14

you can check that using :

curl -XGET "http://localhost:9200/<your_index_name>/_settings"

output will be something like:

{"twitter1":{"settings":{"index":{"codec":"best_compression","number_of_shards":"1","provided_name":"twitter1","creation_date":"1637835561774","number_of_replicas":"1","uuid":"VqwDV_DlRVy3n2iYYnc_3Q","version":{"created":"7060199"}}}}}

15

image
image1128×88 5.59 KB

It is showing this.
Is it apply compression on running index?

16

please read this: i already explained it here

17

Thanks
How can i verify in my output "codec":" best_compression" is not shown.

18

That means the best compression is not yet applied to your index, when did you create the graylog_0 index?

  1. Did you create a template and then create the index graylog_0?
  2. if the answer to point 1 is No then, did you close the index graylog_0 , apply best compression and then reopen the index?
19
  1. I did not created template.
  2. I closed index from gui and then applied compression command and reopened the index.
20

Not sure how you did it, but this is what you should have done for your existing index.

  1. curl -XPOST "http://localhost:9200/graylog_0/_close?pretty"

  2. curl -XPUT 'http://localhost:9200/graylog_0/_settings' -H 'Content-Type: application/json' -d '{"index":{"codec":"best_compression"}}'

  3. curl -X POST "http://localhost:9200/graylog_0/_open?pretty"

Doing it for the template will make sure any new index which gets created using the template will by default have the best compression enabled!

command for that would be : curl -XPUT "http://localhost:9200/_template/my_template" -H 'Content-Type: application/json' -d'{ "index_patterns": ["*"], "settings": { "index": { "codec": "best_compression" } }}'

Hope this helps!