warning: those are probably some newbie questions.
Context: wanting to filebeat some files to ES with ingest pipelines, and IIS logs, from differents files on the same machine.
Are the main configuration from filebeat.yml, and config from IIS module, cleanly isolated ? (example: output.elasticsearch.pipeline and output.elasticsearch.index)
Do I have to make a custom module, one per application ? Is it recommended ?
Modules can be considered as isolated, separated threads ? Is the main config file, can be considered as a module in itself ?
Output configuration is the same for all the inputs and modules in the same filebeat instance. If you want two different outputs, you need two filebeat instances, but this is usually not needed. A single elasticsearch output can send the events to different indexes depending on their content, but this is also usually not needed.
Not sure of understanding this question. There are many modules available, you can use them, for example there is one for IIS, you could use it for your IIS logs, no need to create a custom module, but you may need to customize the configuration for your deployment.
You can also use inputs directly to collect logs from files or other origins. For custom parsing you can use processors or ingest pipelines.
Module files can contain many modules configurations. The main configuration file can contain inputs and modules. Modules configure inputs under the hood, they work mostly independently, each one harvesting their logs, but this is more an implementation detail.
From what you say I believe that all modules share the same output configuration.
In order to understand well, the IIS module creates in ES a pipeline named 'filebeat-7.6.2-iis-access-default'. Where is this output configured in the module ?
And if two modules are enabled, what is the merged output config ?
Each event can contain some metadata about what pipeline should be used to be processed. Modules fill this metadata so the correct pipeline is used when ES receives it. So the output is the same for all modules and inputs, but each event can be processed with a different pipeline.
There is no merged output config, modules or inputs don't modify the output configuration. But they can include in their events some metadata so Elasticsearch knows what pipeline to use for them.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
