I'm new elasticsearch user and i'm in charge to develop a system to monitoring our customers logs. So starting by this point i have to monitoring Fortinet Firewall Logs and i've already did with filebeat sending directly to elasticsearch. I have one index with 3 or 4 customers sending logs simultaneoustly and i need to split them to see their log size by customer. Someone can help me, please ? Or maybe a way to query via console
Welcome to our community!
What version of Elasticsearch are you using?
Hi @YannKistenmacker Also Welcome to the community.
It all comes down to how precise you need to be. It's very simple to just create a percentage of the total logs by customer as a count by customer terms query or Lens visualization and then that's their percentage of the total storage / cost..
It can be that simple if you want.
Firewall logs tend to even out in size across the landscape.
So if that's good enough, no need to even break up the index. It'll save you a lot of other trouble
If you need to know the exact bytes ( which in the end may give you some tiny percentage of precision better) than you will have to break up into separate indices.
Based on your answer i don't need to get the exact amount of data, a percentage's sufficient for me. Can you show me how can i get this percentage without split indices ?
Couple questions how are you collecting the logs?
Are you parsing them?
What fields do you have that have the customer field?
I mean it could be as simple as a Pie Chart
Kibana -> Visualization -> Lens -> Pie Chart
This is a simple example This is my count of records by App.Name yours would be by your customer field.
My infraestructure: Filebeat sends log directly to elasticsearch and parsing the data with "Fortinet Module". I have one field that brings info about customers, in this case "observer.name" field where i can see the name of the customer firewall present in all log entries. I already did a pie with percentage like you, but i would like to know if there is a way to put it in KB/MB/GB form instead percentage.
Ok Yes but it is not super clean... and requires a manual steps
In general your will calculate the avg bytes / log then create a visualization that can show overall storage
So First run this command in Kibana - Dev Tools the
bytes=b means it will show storage in bytes this is sorted by largest but you can also just
s=index to sort by index
Example My Results
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .ds-filebeat-8.2.3-2022.08.06-000146 e1us_WnaRJa26jlYejAxNA 3 1 11447320 0 12206153071 6103214925
green open .ds-filebeat-8.2.3-2022.08.07-000149 Z51Vz3VWRRCrmoXJVAajPg 3 1 11433071 0 12194502328 6096678843
So then do the math... the
store.size is the size of the primary plus what ever replicas... You can use either but the total storage is
so for me
Avg Bytes / Doc = 12206153071 bytes / 11447320 Docs = 1066 Bytes / Doc for prim+replicas
(~500bytes/log primary is very typical)
So then lets make a Lens Table I am basing my on App Name but you can use Customer
This is a table but same for Pie Chart etc..etc..
NOTE : This is for the Time period of the Time Picker so keep that in mind
Then in the Rows the App Name
Then in The Metric choose formula it will even automatically format for Bytes simply
count() * 1066 (my avg bytes per log)
Hope that helps
Okk! I'm going to try to apply this solution today, but looking at it, it seems to me that it will work that way. I will return when i apply these steps.
@stephenb I would like to thank you for all the support, this worked perfectly for me!!! Thank you!!!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.