Need Working ILM example

Elastic Stack Elasticsearch
1

I have tried without success to get ILM working. Can someone post a working index template, rollover alias config, ILM config and anything else I am missing? It seems if you have one of these components incorrect ILM won't work and it won't roll data to warm nodes correctly.

We can tweak times later but would like data to move to warm nodes after 1 days, delete after 3 days for testing.

We can modify days and index sizes once we get everything working.

A working Filebeat config for 7.3 would be perfect.

Thanks
PL

2

Hi PL,
When you deployed Filebeat 7.3 the template, alias, and an ILM policy should have been created, can you tell me:

  1. Did the default policy get created?
  2. If the policy did get created and you added a warm phase: Can you paste in the output of these two Dev Tool -> console commands?

GET /_cat/shards/fileb*?h=index,node

GET /_cat/nodeattrs?v&h=name,attr.name,attr.value

Here is my policy:

    {
    "policy": {
        "phases": {
            "hot": {
                "min_age": "0ms",
                "actions": {
                    "rollover": {
                        "max_age": "1h",
                        "max_size": "50mb",
                        "max_docs": 100
                    }
                }
            },
            "warm": {
                "min_age": "1h",
                "actions": {
                    "allocate": {
                        "include": {},
                        "exclude": {},
                        "require": {
                            "data": "warm"
                        }
                    }
                }
            }
        }
    }
}

Regards,
Dan

3

If you want to try out a fully automated demo, you can pull the code from https://github.com/xeraa/scale-elasticsearch and do a docker-compose up. Let it generate data for 20min and then you'll have a 3 node cluster (1 hot, 1 warm, 1 cold) with data being moved around automatically. For the demo effect if only keeps data for minutes in each phase, but you should be able to adapt that easily.

Also https://github.com/xeraa/scale-elasticsearch/blob/master/setup/setup.sh shows how this is all set up and held together.

2 Likes
4

H is hot nodes W is warm nodes

filebeat-7.3.0 SERVERhio02
filebeat-7.3.0 SERVERhio04
filebeat-7.3.0 SERVERwio03
filebeat-7.3.0 SERVERhio05
filebeat-7.3.0 SERVERwio04
filebeat-7.3.0 SERVERwio01
filebeat-7.3.0 SERVERwio05
filebeat-7.3.0 SERVERhio03
filebeat-7.3.0 SERVERwio02
filebeat-7.3.0 SERVERhio01

{
  "filebeat-7.3.0" : {
    "version" : 9,
    "modified_date" : "2019-09-06T18:16:38.891Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "10mb",
              "max_age" : "1d"
            }
          }
        },
        "warm" : {
          "min_age" : "1d",
          "actions" : {
            "allocate" : {
              "include" : { },
              "exclude" : { },
              "require" : {
                "box_type" : "warm"
              }
            }
          }
        }
      }
    }
  }
}
5

Hi PL,
I expected to see the values "hot" and "warm" as attributes in the output of GET /_cat/nodeattrs?v&h=name,attr.name,attr.value. Did you show that output?

6

Here is that output

Is there a way to just reload the default filebeat ILM config ?

name             attr.name         attr.value
SERVERhio01 ml.machine_memory 67368890368
SERVERhio01 ml.max_open_jobs  20
SERVERhio01 xpack.installed   true
SERVERhio01 box_type          hot
SERVERhio04 ml.machine_memory 67368890368
SERVERhio04 ml.max_open_jobs  20
SERVERhio04 xpack.installed   true
SERVERhio04 box_type          hot
SERVERmst02 ml.machine_memory 8182042624
SERVERmst02 ml.max_open_jobs  20
SERVERmst02 xpack.installed   true
SERVERhio05 ml.machine_memory 67368890368
SERVERhio05 ml.max_open_jobs  20
SERVERhio05 xpack.installed   true
SERVERhio05 box_type          hot
SERVERhio02 ml.machine_memory 67368890368
SERVERhio02 ml.max_open_jobs  20
SERVERhio02 xpack.installed   true
SERVERhio02 box_type          hot
SERVERhio03 ml.machine_memory 67368890368
SERVERhio03 ml.max_open_jobs  20
SERVERhio03 xpack.installed   true
SERVERhio03 box_type          hot
SERVERwio02 ml.machine_memory 67368890368
SERVERwio02 ml.max_open_jobs  20
SERVERwio02 xpack.installed   true
SERVERwio02 box_type          warm
SERVERmst01 ml.machine_memory 8182050816
SERVERmst01 ml.max_open_jobs  20
SERVERmst01 xpack.installed   true
SERVERmst03 ml.machine_memory 8182042624
SERVERmst03 ml.max_open_jobs  20
SERVERmst03 xpack.installed   true
SERVERwio01 ml.machine_memory 67368890368
SERVERwio01 ml.max_open_jobs  20
SERVERwio01 xpack.installed   true
SERVERwio01 box_type          warm
SERVERwio03 ml.machine_memory 67368890368
SERVERwio03 ml.max_open_jobs  20
SERVERwio03 xpack.installed   true
SERVERwio03 box_type          warm
SERVERwio05 ml.machine_memory 67368890368
SERVERwio05 ml.max_open_jobs  20
SERVERwio05 xpack.installed   true
SERVERwio05 box_type          warm
SERVERwio04 ml.machine_memory 67368890368
SERVERwio04 ml.max_open_jobs  20
SERVERwio04 xpack.installed   true
SERVERwio04 box_type          warm
SERVERkab01 ml.machine_memory 8182042624
SERVERkab01 ml.max_open_jobs  20
SERVERkab01 xpack.installed   true
7

I can ask about the process for removing and reinstalling the policy, but before you do that can you try this please:

GET /_cat/shards/fileb*?h=index,node

8

@P_Larsen

Let me know how you get along. I would love to see the output of GET /_cat/shards/fileb*?h=index,node before you remove and replace the policy.

To remove and replace the filebeat ILM policy you have to:

  • Remove the policy from any existing indices
  • Delete the policy
  • Re-run filebeat setup

Remove the policy from existing indices:

  • Go into Index Management and change the Rows per page at bottom to 100
  • type in filebeat in the search bar
  • tick the box above the index list to select all
  • click on manage indices and Delete Policy (I forget the exact text, but I think it is at bottom)
  • repeat for as many pages of indices you have

Delete the policy:

  • From the ILM UI click on Actions for the policy and Delete policy

Run setup:
./filebeat setup --ilm-policy -v

9

Thanks for the details I will work on getting filebeat reset again.

This appears to be a good working example except the "Creating the first index" step has a typo.

Change
`"logstash-eagleeye-brofiler"` `: {`
to
`"logstash-eagleeye-brofilter"` `: {`

Other lessons learned:
Logstash needs to send data to the index alias.

I reduced the settings down to a couple of days so will test using that to see if the data rolls.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.