The aggregate filter is working great in my testing, but I'm concerned about scale because it is stateful and requires setting filter workers to 1 (-w 1 flag). We have about 150 servers, each of which will generate ~2GB of logs per day. The log structure is generally like this:
sameTaskId, start, username,size sameTaskId,networkstats anotherTaskId, start, username,size sameTaskId, end, authAuditInfo anotherTaskId,networkstats anotherTaskId, end, authAuditInfo
We love the idea of aggregating in order to reduce storage costs and to avoid aggregating in every query, but are worried about 1 worker keeping up in real-time. How can we solve this? Based on my research, I have a few starter ideas, none of which sound ideal:
- Static load balancing rules - We could have each Logstash instance behind a load balancer configured to always route server traffic to the same instance. Cons: we would need to manage a large Logstash cluster and this kind of load balancing is not resilient to failures or needing to add/remove machines.
- Use pipeline-to-pipeline communication to fan out to multiple pipelines per machine as described here. Cons: could increase throughput of a machine, but would probably still need the static load balancing described above.
- Instead of the aggregate filter, use an entity-centric index as described here. I need to better understand this, but I'm concerned about the comment that the data would also be stored in the standard index. Also, I'm not sure where in the pipeline this aggregating would occur. During ingestion?
- Instead of the aggregate filter, use a batch process to periodically aggregate data after it is in Elasticsearch as described here. I'm open to this, but would need an example of how this could be done. Would this use the experimental "Rollup" feature or the "Transforms" feature? If not, would we have to use the API or a client SDK?
At the end of the day, I'd love to aggregate our logs, but want to be sure we can scale and observe system behavior in real-time. Thanks in advance for the input!