I need to collect data from several hundred different types of logs, each with a slightly different format from each other (organic growth of logging formats). I would prefer to institute a separate pipeline for each type of log, but I'm worried about logstash's resource consumption and performance if I do so.
Scaling due to amount of data ingested is a separate issue, I'm happy to create multiple logstash servers, but that only works if logstash handles loaded but unused pipelines sparsely.
Does logstash statically allocate resources per pipeline? Or can I add a near unlimited number of pipelines to a single server as long as I don't send too much traffic to it at any given time?