I need some advice on my design of indices / types in the following context :
I have set up a single (for, now, I will maybe add nodes later) ElasticSearch server for several purposes :
- In Kibana, be able to search the logs of multiple applications gathered from multiple servers with FileBeat. Those logs are, for the most part, parsed into structured documents with LogStash
- In Kibana, be able to produce vizualizations and dashboards from said structured data parsed with LogStash and Metricbeat for those applications.
- In Kibana, produce Metrics dashboards from data gathered by MetricBeat
I have 3 Applications A, B and C with different kinds of logs for each. As a consequnce, I have "mapped" the log kinds to ElasticSearch types, for instance :
- A has types A1, A2 and A3
- B has types B1, B2 and B3
- C has types C1, C2 and C3
Those types all have different mappings which chare very few fields (basically only the timestamp and filebeat default fields) :
I have chosen the following indices /types setup :
- Application A logs go to daily rolling (via Logstash) indices called A-YYYY.mm.dd
- Application B logs go to daily rolling (via Logstash) indices called B-YYYY.mm.dd
- Application C logs go to daily rolling (via Logstash) indices called C-YYYY.mm.dd
I also have setup index aliases so that alias A points to A* indices, same for B and C, obviously.
The part where it gets tricky is that I want to keep some information longer than other :
Than means, for instance, that I want to delete all type A1 documents older than 7 days from A indices (metricbeat metricsets for instance) but I want to keep A3 type documents for at least 300 days for legal reasons.
I was thinking about using curator for this kind of maintainance but realized that curator was only acting on indices and does not go allow to go down to the document type level.
I then thought about using the delete by query API on the aliases but have read that it was a very bad practice.
My question is simple :
Is my setup suitable for what I want to achieve here or do I have to design all this differently so that types are spread accross their own indices even though it will create hundreds if not thousands of indices over a period of more that 1 year ?
Thanks a lot in advance for your help