Here is an overview of what I am trying to do. I have my Logstash config file set up to index like this index => "%{[@metadata][beat]}-%{+yyyy.ww}". When I try to load the winlogbeat beat template manually following the instruction and load the dashboard it loads winlogbeat-7.3.0-(yyyy.mm.dd) and the dashboards don't load. Is there a way to change the template to use the format my Logstash is set to use? I hope that is clear.
thanks in advance.
perhaps the logstash config should look below ... looks like you are missing the version portion of the index name ... See Docs Here
output {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}I really want to keep it yyyy.ww.
Weekly is fine I think you are missing the version portion if you want that... %{[@metadata][version]}
The visualizations should be using winlogbeat-* as the index pattern so it really should not matter.
Which dashboards? Which modules are you using?
The default dashboard for Winlogbeat.
it should be using winlogbeat-* as the index pattern.
Can you see the data in discover?
It seems not to like the yyyy.ww with the version number.
did you still run .\winlogbeat.exe setup
I always configure winlogbeat to point directly to the cluster first and then run .\winlogbeat.exe setup .... that loads all the index templates dashboards etc.. etc..
Then I move on to configure with logstash ....
if you just loaded the template then the dashboards may not be loaded,
or run .\winlogbeat.exe setup --dashboards to setup the dashboards I would run .\winlogbeat.exe setup
I run through the winlogbeat setup and loaded everything manually. After I restarted all 3 services it is working now. I am not sure I did to get it to work. I do thank you for all your help.
For my understanding the dashboard use the winlogbeat-* to populate? It doesn't matter how the indices is used either (yyyy.mm.dd) or (yyyy.ww)?
Cool
Yup as long as the index starts with winlogbeat-*
I would assume that also holds true with the other beats then?
Yes it does
Thank you for all your help. It has been much appreciated.
I do have one more question for my understanding. Before I made the change to the logstash config to add the version number to the index why wouldn't everything work without that? The index was still winlogbeat-*. The indices were winlogbeat-(yyyy.ww).
I don't know ... it should have... assuming everything else was set up correctly ... it seems perhaps one of the setup items was not loaded or not loaded properly
Now I am receiving indice lifecycles errors. I think it is related to what I did yesterday to fix index name. How can I resolve this? Below is the error I'm receiving and from what I can tell it is an issue with the alias.
{
"indices" : {
"winlogbeat-7.3.0-2019.33" : {
"index" : "winlogbeat-7.3.0-2019.33",
"managed" : true,
"policy" : "winlogbeat-7.3.0",
"lifecycle_date_millis" : 1565724724836,
"phase" : "hot",
"phase_time_millis" : 1565807461159,
"action" : "rollover",
"action_time_millis" : 1565725464982,
"step" : "ERROR",
"step_time_millis" : 1565807948300,
"failed_step" : "check-rollover-ready",
"step_info" : {
"type" : "illegal_argument_exception",
"reason" : "index.lifecycle.rollover_alias [winlogbeat-7.3.0] does not point to index [winlogbeat-7.3.0-2019.33]",
"stack_trace" : """
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [winlogbeat-7.3.0] does not point to index [winlogbeat-7.3.0-2019.33]
at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:835)
"""
},
"phase_execution" : {
"policy" : "winlogbeat-7.3.0",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d"
}
}
},
"version" : 14,
"modified_date_in_millis" : 1565724172067
}
}
}
}When are you receiving that error?
What is that code above the result of which command?
Check the winlogbeat template and see how it is set, do you have more than one template for winlogbeat?
GET _template/winlogbeat-7.3.0
I got that from the below code. I have been reading up on the ILM policy and how to create them. From what I understand it is having trouble matching the ILM policy and the indice together. I have also tried creating a hot-warm-cold-delete policy and it is saying "No node attributes configured in elasticsearch.yml
You can't control shard allocation without node attributes.".
GET /winlogbeat-7.3.0-2019.33/_ilm/explain
My biggest confusion is trying to associate the policy with the indice and index pattern.
Thank you
That is associated with the template I believe...
Have you read
Yes, I have read that article. When creating the policy I don't understand which names to use where. I have confused myself. Below is the policy I am trying to use and not sure why I can't get it to associate with the indices.
Error Message: "type" : "illegal_argument_exception",
"reason" : "index.lifecycle.rollover_alias [winlogbeat-7.3.0] does not point to index [winlogbeat-7.3.0-2019.33-1]",
"stack_trace" : """
Index Pattern: winlogbeat-*
Indices: winlogbeat-7.3.0-2019.33-1
Alias: not sure what to name it.
PUT /_ilm/policy/winlogbeat-7.3.0
{
"policy": {
"phases": {
"hot": {
"actions": {
"rollover": {
"max_size": "10GB",
"max_age": "30d"
}
}
},
"delete": {
"min_age": "90d",
"actions": {
"delete": {}
}
}
}
}
}
PUT /_template/winlogbeat_template
{
"index_patterns": ["winlogbeat-*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"index.lifecycle.name": "winlogbeat-7.3.0",
"index.lifecycle.rollover_alias": "winlogbeat-7.3.0"
}
}
PUT /winlogbeat-7.3.0-2019.33-1
{
"aliases": {
"winlogbeat-": {
"is_write_index": true
}
}
}© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.