Forward before processing

[criddock (Cary Riddock)]

Greetings and Happy 4th to Everyone...

My question is this... I have a customer that wants to forward a copy of all received event traffic to a cold storage server before Logstash can process the event...We are doing some GROK and KV work on the events and doing some other mods. He wants the original events preserved as received to keep aligned with his data retention mandates.

Based on what I can tell, there is no native mechanism to do this since the OUTPUTS section follows the FILTER directives section...I tried creating just a CONF that just forwards with no processing, but run afoul of duplicate INPUTS configuration.

Any thoughts would be appreciated.

(( If there was a PRIORITY.CONF file that outlined which CONF files and in which order they should be executed, that would be SUPER useful))

[Badger]

I would do this using pipelines. In non-production I might even use the beta pipeline-to-pipeline communication.

Alternatively, if you are on an old version that does not support pipelines, set up a second logstash instance. Have the first one consume the message from the origin and send it to two outputs. One would be the cold storage server, the other would be a tcp output on localhost. Then the existing logstash process would consume a tcp input on localhost.