Please find the details about filebeat.yml in the below link.
1 Like
Ohhh you are going through logstash...
You did not mention that... do you need logstash? its adds complexity... and is why your ILM is not working
Its OK but this means you have an issue there...
Share your logstash conf file... particularly the output.
Or take out logstash and go directly to Filebeat -> Elasticsearch
Anyway... your old indices will never roll over and get deleted because they were not set up correctly because you put logstash in the middle and did not configure it correctly ...
You need to run
./filebeat setup -e
When filebeat is pointing to elasticsearch
Then change Filebeat to point to logstash and then use the correct config
AND yes the docs are wrong (I thought I had them fix it)
it should look like (notice the index name)
So you probably followed the docs bu they are not correct.
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
pipeline => "%{[@metadata][pipeline]}"
user => "elastic"
password => "secret"
}
} else {
elasticsearch {
hosts => "https://061ab24010a2482e9d64729fdb0fd93a.us-east-1.aws.found.io:9243"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
user => "elastic"
password => "secret"
}
}
}
1 Like
cat 30-elasticsearch-output.conf
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
1 Like
Should be .... the following... if you want ILM to work.
Your logstash config is wrong it is meant for daily indices not ILM.
This should work going forward... your other indices will never roll over because they are "Daily Indices" and are not using the Write Alias which is
filebeat-7.17.13
hence
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
Correct
output {
if [@metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
pipeline => "%{[@metadata][pipeline]}"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
}
You will need to manually delete the old indices as they will never rollover and thus never move to the next step and never be deleted
To be direct the Guide you are following was intended to use Daily Indices not ILM... until now .. we did not know that... and that is the source of most of these issues.
You still need to run
./filebeat setup -e
At least once while filebeat is pointing to Elasticsearch BEFORE you start sending data through logstash.
See it should look like this you are missing all the write alias ....
GET /filebeat-7.17.3-2024.01.09-000001
{
"filebeat-7.17.3-2024.01.09-000001" : {
"aliases" : { <!--- THIS
"filebeat-7.17.3" : { <!--- THIS
"is_write_index" : true <!--- THIS
}
},
1 Like
Thanks, Stephen.
I will check and update on my findings.
1 Like
Hi @stephenb
I have a query on the below and this is specific to production environment.
-
Does ILM supported on the basic version of ELK (7.17.15)?
-
Does the attached in the below URL is also having issues? Because I am seeing exceptions when I run GET /filebeat-7.17.15-2024.01.10/_ilm/explain.
-
The existing production setup has implementation as filebeat > Logstash > Elasticsearch
And taking out logstash to make ILM work may not be feasible at the moment. -
Since the indices are piling up on daily basis result in huge disk utilization. do we have a solution on how this can sorted out? Please suggest.
Thanks,
Yes Absolutely
That is EXACTLY what I am trying to explain ... you have Setup ILM vs Daily Indices and are mixing the concept ... and this is causing the errors.... you following a 3rd party doc / config which does NOT support what you want to do...
But It can be easily fixed if you follow my instructions.
That is fine I gave you the solution above... with the output section that will fix your ILM issue for indices going forward it will NOT fix the old broken indices.
Replace your output section with what I gave you AND ILM should start working
BUT you need to run
filebeat setup -e
FIRST before you send the data through logstash
Then your ILM will work as you want.
Can you run
GET _alias/filebeat*
And show the result please
This can be fixed but you actually need to do what I am suggesting otherwise it will never work
You will need to delete them manually... which is easy to do using
DELETE indexname
WARNING that will permanently delete the indices and the data will be lost
Thanks, Stephen for your quick response. Please find the details below.
{
"filebeat-7.17.15-2024.01.05-000003" : {
"aliases" : {
"filebeat-7.17.15" : {
"is_write_index" : true
}
}
},
"filebeat-7.17.15-2023.12.29-000002" : {
"aliases" : {
"filebeat-7.17.15" : {
"is_write_index" : false
}
}
}
}
Hi Stephen
Another thing that I have prepared the procedural doc on applying the changes as per your suggestions on the test instance.
Please review the doc from the link below and please let me know if its correct so that I can start applying changes.
After running the filebeat setup -e. do I need to revert the changes in filebeat.yml again i.e. pointing filbeat -> Logstash
Thanks,
This is good it means you have a viable write alias...
Hi Ravi... quick look ... not quite right...
Steps Should be
- Fix Logstash output
- Point filebeat output to elasticsearch
./filebeat setup -e(your command is not correct)- Point filebeat output back to logstash
- Start Filebeat
You will need to test and try... I am not really have time to review all your procedures...
Hi Stephen
I have run the procedure and I could see as below when compared to yours in previous post.
Latest from my test instance as below.
GET filebeat-7.17.13-2024.01.09-000006
{
"filebeat-7.17.13-2024.01.09-000006" : {
"aliases" : {
"filebeat-7.17.13" : {
"is_write_index" : true
}
},
So now if you configure logstash output as I suggested...
Logstash will write data to the write alias filebeat-7.17.13 (which points to the concrete index), and your ILM, rollover and delete should work per the policy...
In order for ILM to work... the data needs to be written to the write alias...
I would also suggest to run this.
POST filebeat-7.17.3/_rollover
That will force the write alias to to rollover the index, and the new index will follow your new ILM Policy, and you should be good to go!
POST filebeat-7.17.13/_rollover
{
"acknowledged" : true,
"shards_acknowledged" : true,
"old_index" : "filebeat-7.17.13-2024.01.11-000007",
"new_index" : "filebeat-7.17.13-2024.01.11-000008",
"rolled_over" : true,
"dry_run" : false,
"conditions" : { }
}
You should be good to go... Looks like a new index what just created today anyways but that should not matter...
Start Logstash with the Config
Point Filebeat to Logstash
And you should see data flowing into filebeat-7.17.13-2024.01.11-000008
This was done. reference output below.
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
You mean as below..
root@ELK-test:/usr/share/logstash# bin/logstash -f /etc/logstash/conf.d/30-elasticsearch-output.conf
But it give some errors. file attached in the below URL.
You need to start logstash the normal way... probably
systemctl start logstash
I feel like I am helping in the dark.....
I am not following your guide (you have not shared it... but also we recommend following our documentation) so I have no clue how you installed logstash and what you did with logstash etc.etc...
If you have questions on starting / stopping running logstash etc... you should open a separate thread on that...
yeah, I have restarted the LS service and now its up and running.
# systemctl status logstash
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2024-01-11 20:52:15 UTC; 2s ago
Main PID: 5293 (java)
Tasks: 14 (limit: 4667)
I have followed the guide from the sources on the installation of ELK stack.
would request you to share your documentation which will be of a great help.
Thanks,
So that "HowTo" looks like it was pretty good... but it is ~3.5+ years old... Elastic moves fast so that is an issue... and why the new ILM did not work with the old config etc..
There are many many resources ...
I am sure you can look around you are doing fine... just keep working and asking questions
Good Luck!
Thanks, Stephen for your guidance and support on this thread.
I will keep posted on my observations on post changes today.
Hi Stephen,
ILM is working as expected in the test environment and thanks for your guidance on the same.
I am planning to implement the same in the production system today. Currently I came across with the below observations.
GET /filebeat-8.10.4-2024.01.12
{
"filebeat-8.10.4-2024.01.12" : {
"aliases" : { },
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
GET /filebeat-8.11.2-2024.01.12
{
"filebeat-8.11.2-2024.01.12" : {
"aliases" : { },
"mappings" : {
"properties" : {
"@timestamp" : {
"type" : "date"
},
Also w.r.t 7.7.15 for which I am planning to run the steps for ILM the GET request shows as below. And I can handle this case as I have did this in test bed yesterday only.
GET /<filebeat-7.17.15>
{
"filebeat-7.17.15-2024.01.05-000003" : {
"aliases" : {
"filebeat-7.17.15" : {
"is_write_index" : false
}
so, in these conditions mentioned above for index 8.10 and 8.11. How to handle when it comes to rollover e.g. POST filebeat-7.17.3/_rollover
Thanks,