I have written a conf file to collect log file data about cpu usage. Is there anyway to do prediction of my cpu usage in the future 30 days with my current cpu usage data in the logstash conf file?
Well, Logstash certainly won't predict future values. Has anything in the environment changed since last month? If not, why wouldn't the best prediction of the CPU usage for month N+1 be the CPU usage of month N?
How about if I get a manual equation to make the prediction for that, then will there be a possible way of doing that with logstash or elasticsearch?
I'm not sure exactly what you mean by "manual equation", but neither Logstash nor Elasticsearch are equation solvers. Logstash is a simple data processing pipeline and ES is a search engine.
I have found the moving average function which can be used in kibana. How about if using moving average function to predict future value?
Sure, that could work. Elasticsearch's moving average aggregation documentation discusses this to some degree.
But as from my query
.es(index='linux_cpu-*', metric='avg:CPU(%)').movingaverage(10)
It does not seem to work properly.
This can happen if the moving average model doesn't match your data or if the choice of parameters isn't optimal. In this particular case maybe the window size is too small. This is not my area of expertise so I won't be able to offer any more help.
I have found from Moving average aggregation | Elasticsearch Guide [8.11] | Elastic which states that moving average can do predictions. But I am not sure how should I do that.
Should I just include the following into the logstash output json file for the related index as an output mapping?
{
"the_movavg":{
"moving_avg":{
"buckets_path": "the_sum",
"window" : 30,
"model" : "simple",
"predict" : 10
}
}
This is my json file for logstash
{
"template" : "linux_cpu-",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"default" : {
"_all" : {"enabled" : true, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@timestamp": { "type": "date" },
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"dynamic": true,
"properties" : {
"ip": { "type": "ip" },
"location" : { "type" : "geo_point" },
"latitude" : { "type" : "float" },
"longitude" : { "type" : "float" }
}
}
}
}
}
}