Trying to reduce the storage used by elasticsearch2.2

Elastic Stack Elasticsearch
1

I am trying to reduce the storage used by elasticsearch. After reading the options available.
Looks like compression is by default enabled in elasticsearch 2.2.
But we can change the default to "best_compression".
When I am trying to run this command. It throws me an error.
I am trying to run this in different ways like

  1. curl -XPUT ‘localhost:9200/twitter/_settings’ -d ‘{“index”: {“codec”: “best_compression”}}’
  2. curl -XPOST ‘localhost:9200/twitter’ -d ‘{“settings”: {“index.codec”: “best_compression”}}’
    I am running this from terminal "Workstation:~/ELK/elasticsearch-2.2.0$"

Getting the following error
"""
curl: (6) Could not resolve host: xn--localhost-499d
curl: (3) [globbing] unmatched brace in column 1
curl: (3) [globbing] unmatched close brace/bracket in column 23
"""

Can anyone help me on this?

2

This works in Sense ok (but complains cause the index is open);

PUT au-disaster-events/_settings
{
  "index":
  {
    "codec": "best_compression"
  }
}

But when I try via curl it doesn't like it;

$ curl -XPUT "localhost:9220/au-disaster-events/_settings" -d "{
  "index":
  {
    "codec": "best_compression"
  }
}"
{"error":{"root_cause":[{"type":"settings_exception","reason":"Failed to load settings from [{\n  index:\n  {\n    codec: best_compression\n  }\n}]"}],"type":"settings_exception","reason":"Failed to load settings from [{\n  index:\n  {\n    codec: best_compression\n  }\n}]","caused_by":{"type":"json_parse_exception","reason":"Unrecognized token 'best_compression': was expecting ('true', 'false' or 'null')\n at [Source: {\n  index:\n  {\n    codec: best_compression\n  }\n}; line: 4, column: 28]"}},"status":500}

I'm just about to get on a plane (no internet), but I'll come back to this.

3

curl: (6) Could not resolve host: xn--localhost-499d

It looks like Microsoft Office or similar quote-damaged your command. Make sure the quotes are plain ASCII straight quotes. Comparison between your command and what it should look like:

curl -XPUT ‘localhost:9200/twitter/_settings’ -d ‘{“index”: {“codec”: “best_compression”}}’
curl -XPUT 'localhost:9200/twitter/_settings' -d '{"index": {"codec": "best_compression"}}'
4

Awesome.. it worked.

Thanks a ton @magnusbaeck

5

Can I somehow put these settings in the elasticsearch.yml or elasticsearch.cfg file? So that everytime I create a new index, I don't have to do it manually.

Also, regarding the types of various fields, by default it takes it as "string". I can specify the types using templates, but again, can I automate for everytime I add a new node in the cluster?

6

Can I somehow put these settings in the elasticsearch.yml or elasticsearch.cfg file? So that everytime I create a new index, I don't have to do it manually.

Use an index template.

Also, regarding the types of various fields, by default it takes it as "string". I can specify the types using templates, but again, can I automate for everytime I add a new node in the cluster?

Index templates are cluster-wide so there's nothing to automate.

(This isn't true for index templates placed in the file system of each node, but that method was removed in ES 2.0.)

7

Okay, but suppose I have multiple clusters. Can I use templates in some configuration file?

8

Okay, but suppose I have multiple clusters. Can I use templates in some configuration file?

As I said, that feature was deprecated in ES 2.0. You'll have to post the index templates via the API. That's easy to automate.