I have a design question. Let's say I have 1 filebeat process to monitor X log-files. The log-data is sent to ES through a pipeline. The pipeline is anyway required for date/timestamp processing.
The data when stored in ES has to be enriched with some extra meta-data in the form of extra document fields and tags.
I see two approaches to accomplish this enrichment:
(1) Filebeat with 1 prospector per file making it possible to add the extra fields and tags immediately in the prospector configuration (hardcoding). And thus, much less work to be done in the pipeline.
(2) Filebeat with 1 prospector for all files, but then having a pipeline doing more work (aka. grok pattern matching) to construct the extra fields and tags on the fly.
Which of the two designs would the most optimal ?