Hi, Im planning the way to organizing my data collection in terms of indexing and organizing flows in some way. Im goin to collect large amounts of data of 3 main sorts - system logs, application/proxy/loadbalncer logs and firewall/network logs.
The way i want to work with this is to use filebeat modules (ngnix, apache, cisco, juniper etc) for most of them (and logstash/ingest pipelines for specific patterns). I assume it may be 40-80GB data (daily) from all sources for a start. I will probably use 10-20 filebeats. Of course I am planning to use ILM and aliases with ECS scheme.
The question is how to plan the way I`m indexing data?
- Should I divide data into multiple indexes basing on "category" of data (system/app/network):
index: "filebeat-%{[agent.version]}-systemlogs-%{+yyyy.MM.dd}"
index: "filebeat-%{[agent.version]}-network-%{+yyyy.MM.dd}"
index: "filebeat-%{[agent.version]}-app-%{+yyyy.MM.dd}"
- Or maybe distinguish by module as well (what will be usefull in some aggregations/queries later):
index: "filebeat-%{[agent.version]}-systemlogs-%{[event.module]}-%{+yyyy.MM.dd}"
index: "filebeat-%{[agent.version]}-network-%{[event.module]}-%{+yyyy.MM.dd}"
index: "filebeat-%{[agent.version]}-app-%{[event.module]}-%{+yyyy.MM.dd}"
- Data sources come from few regions/locations (up to 20GB/day per location) - I`m planning on putting some tag/additional field or should split indexes basing on it?
Maybe there is some more convenient and useful way to handle it?
Data will be used commonly by separate teams (for example firewall flows) that`s why I thought about separating those categories/fields of interest. Of course there will be use-cases for large aggregations combining both loadbalancers and firewalls as well.
I know that there is no one answer for such a question but I`ll be glad for giving some advice on what to take into consideration. Maybe You may tell me what direction should I take in plannig on how to oranize data so that (further) growth will be easier.