P_Larsen
September 6, 2019, 1:13pm
1
I have tried without success to get ILM working. Can someone post a working index template, rollover alias config, ILM config and anything else I am missing? It seems if you have one of these components incorrect ILM won't work and it won't roll data to warm nodes correctly.
We can tweak times later but would like data to move to warm nodes after 1 days, delete after 3 days for testing.
We can modify days and index sizes once we get everything working.
A working Filebeat config for 7.3 would be perfect.
Thanks
Hi PL,
Did the default policy get created?
If the policy did get created and you added a warm phase: Can you paste in the output of these two Dev Tool -> console commands?
GET /_cat/shards/fileb*?h=index,node
GET /_cat/nodeattrs?v&h=name,attr.name,attr.value
Here is my policy:
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1h",
"max_size": "50mb",
"max_docs": 100
}
}
},
"warm": {
"min_age": "1h",
"actions": {
"allocate": {
"include": {},
"exclude": {},
"require": {
"data": "warm"
}
}
}
}
}
}
}
Regards,
xeraa
September 8, 2019, 1:39am
3
If you want to try out a fully automated demo, you can pull the code from https://github.com/xeraa/scale-elasticsearch and do a docker-compose up. Let it generate data for 20min and then you'll have a 3 node cluster (1 hot, 1 warm, 1 cold) with data being moved around automatically. For the demo effect if only keeps data for minutes in each phase, but you should be able to adapt that easily.
Also https://github.com/xeraa/scale-elasticsearch/blob/master/setup/setup.sh shows how this is all set up and held together.
2 Likes
P_Larsen
September 9, 2019, 12:28pm
4
H is hot nodes W is warm nodes
filebeat-7.3.0 SERVERhio02
filebeat-7.3.0 SERVERhio04
filebeat-7.3.0 SERVERwio03
filebeat-7.3.0 SERVERhio05
filebeat-7.3.0 SERVERwio04
filebeat-7.3.0 SERVERwio01
filebeat-7.3.0 SERVERwio05
filebeat-7.3.0 SERVERhio03
filebeat-7.3.0 SERVERwio02
filebeat-7.3.0 SERVERhio01
{
"filebeat-7.3.0" : {
"version" : 9,
"modified_date" : "2019-09-06T18:16:38.891Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "10mb",
"max_age" : "1d"
}
}
},
"warm" : {
"min_age" : "1d",
"actions" : {
"allocate" : {
"include" : { },
"exclude" : { },
"require" : {
"box_type" : "warm"
}
}
}
}
}
}
}
}
Hi PL,GET /_cat/nodeattrs?v&h=name,attr.name,attr.value. Did you show that output?
P_Larsen
September 9, 2019, 7:27pm
6
Here is that output
Is there a way to just reload the default filebeat ILM config ?
name attr.name attr.value
SERVERhio01 ml.machine_memory 67368890368
SERVERhio01 ml.max_open_jobs 20
SERVERhio01 xpack.installed true
SERVERhio01 box_type hot
SERVERhio04 ml.machine_memory 67368890368
SERVERhio04 ml.max_open_jobs 20
SERVERhio04 xpack.installed true
SERVERhio04 box_type hot
SERVERmst02 ml.machine_memory 8182042624
SERVERmst02 ml.max_open_jobs 20
SERVERmst02 xpack.installed true
SERVERhio05 ml.machine_memory 67368890368
SERVERhio05 ml.max_open_jobs 20
SERVERhio05 xpack.installed true
SERVERhio05 box_type hot
SERVERhio02 ml.machine_memory 67368890368
SERVERhio02 ml.max_open_jobs 20
SERVERhio02 xpack.installed true
SERVERhio02 box_type hot
SERVERhio03 ml.machine_memory 67368890368
SERVERhio03 ml.max_open_jobs 20
SERVERhio03 xpack.installed true
SERVERhio03 box_type hot
SERVERwio02 ml.machine_memory 67368890368
SERVERwio02 ml.max_open_jobs 20
SERVERwio02 xpack.installed true
SERVERwio02 box_type warm
SERVERmst01 ml.machine_memory 8182050816
SERVERmst01 ml.max_open_jobs 20
SERVERmst01 xpack.installed true
SERVERmst03 ml.machine_memory 8182042624
SERVERmst03 ml.max_open_jobs 20
SERVERmst03 xpack.installed true
SERVERwio01 ml.machine_memory 67368890368
SERVERwio01 ml.max_open_jobs 20
SERVERwio01 xpack.installed true
SERVERwio01 box_type warm
SERVERwio03 ml.machine_memory 67368890368
SERVERwio03 ml.max_open_jobs 20
SERVERwio03 xpack.installed true
SERVERwio03 box_type warm
SERVERwio05 ml.machine_memory 67368890368
SERVERwio05 ml.max_open_jobs 20
SERVERwio05 xpack.installed true
SERVERwio05 box_type warm
SERVERwio04 ml.machine_memory 67368890368
SERVERwio04 ml.max_open_jobs 20
SERVERwio04 xpack.installed true
SERVERwio04 box_type warm
SERVERkab01 ml.machine_memory 8182042624
SERVERkab01 ml.max_open_jobs 20
SERVERkab01 xpack.installed true
I can ask about the process for removing and reinstalling the policy, but before you do that can you try this please:
GET /_cat/shards/fileb*?h=index,node
DanRoscigno
September 11, 2019, 2:05pm
8
@P_Larsen
Let me know how you get along. I would love to see the output of GET /_cat/shards/fileb*?h=index,node before you remove and replace the policy.
To remove and replace the filebeat ILM policy you have to:
Remove the policy from any existing indices
Delete the policy
Re-run filebeat setup
Remove the policy from existing indices:
Go into Index Management and change the Rows per page at bottom to 100
type in filebeat in the search bar
tick the box above the index list to select all
click on manage indices and Delete Policy (I forget the exact text, but I think it is at bottom)
repeat for as many pages of indices you have
Delete the policy:
From the ILM UI click on Actions for the policy and Delete policy
Run setup:
P_Larsen
September 11, 2019, 3:31pm
9
Thanks for the details I will work on getting filebeat reset again.
This appears to be a good working example except the "Creating the first index" step has a typo.
Change
`"logstash-eagleeye-brofiler"` `: {`
to
`"logstash-eagleeye-brofilter"` `: {`
Other lessons learned:
I reduced the settings down to a couple of days so will test using that to see if the data rolls.
system
October 9, 2019, 3:39pm
10
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.