sorry to bother you. just wondering and by wondering I mean , smashing my head on the computer desk over and over thinking and probably over thinking it ...
Lets say I have access to 1 server 16gigabyte of ram, 4 cpu 3ghz
- NOT TWO, one
I would like to use that ELK stack for trafic monitoring purpose. using log from fortigate device, currentely sent to syslogd server ( same machine as described earlier ) using logstash to forward those logs to ES and... finaly visualise em in Kibana !
Lets also say that I currently have ~ 25g of logs per day created in a "daily indice" ANDDDDD we will more than probably reach ~100gig of logs per day ( per daily indices )
now, to add complexity ! In a near future, we would like to be able to see up to 30 days of "logs information" in a single visualisation dashboard ( currentely at ~25giga we cannot. we get a "freaking 30 000ms timeout" in kibana for only 7days dashboard... ******** any help is welcome here.. ********** any way to generate a view no matter how long it will take ? like a monthly overnight report ? I couldnt care less... as long I can see it when needed...)
now ... ive been thru almost everything to optmize the stack,
desactivate swap in fstab
change refresh interval
Everything I can think of right now OR found on internet, but "personalise my mapping" and honestly, this part.. is absolutely a pain in the .... I am really forced to do it? I mean ffs. I have more than 200 field that would need to be personalised, exclude or whatever else...
AM I REALLY stuck with that customisation to do ? ( not only that but afterward... I would have to figure out a way to reindex the whole thing after creating my personalised template.. )
See my problem is ... it is a permanentely moving machine, It will continue again and again and again to receive logs from FTG device AND index em. it will never stop. So, i cannot stop logstash instance or whatever else to dedicate ressource that might be somehow missing neighter can I stop the "indexation"
Please, help me... Im literally close to lose my mind playing in those "freaking" templates...