Filebeat only supports one output, Logstash supports multiple outputs and is way more flexibel than Filebeat.
To do this in Logstash you will need to use multiple pipelines with the pipelines.yml file and configure it to use the pipeline-to-pipeline communicaton.
You will need these 3 pipelines:
main.conf
input {
your data input
}
output {
pipeline {
send_to => "raw"
}
pipeline {
send_to => "filtered"
}
}
Then you will have the raw.conf and filtered.conf pipelines.
raw.conf
input {
pipeline {
address => "raw"
}
}
output {
elasticsearch {
your elastic configuration for the raw data
}
}
and
filtered.conf
input {
pipeline {
address => "filtered"
}
}
filter {
your filters to parse and enrich your message
}
output {
elasticsearch {
your elastic configuration for the raw data
}
}
Your pipelines.yml will have the following configuration:
- pipeline.id: main
path.config: "/path/to/main.conf"
- pipeline.id: raw
path.config: "/path/to/raw.conf"
- pipeline.id: filtered
path.config: "/path/to/filtered.conf"
Basically you are receiving your data in the main pipeline and send the same message to the other two pipelines where you can apply different filters and send to differente destinations.
But as @Christian_Dahlqvist said, this is not usual since you already can keep the raw message in the same index, keep in mind that this approach will use more disk as you are storing things twice.